| Server IP : 3.111.61.48 / Your IP : 216.73.216.67 Web Server : Apache System : Linux ip-10-0-5-176 6.8.0-1057-aws #60~22.04.1-Ubuntu SMP Wed May 27 08:16:59 UTC 2026 x86_64 User : ubuntu ( 1000) PHP Version : 8.2.31 Disable Function : exec,passthru,shell_exec,system MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : ON Directory : /usr/lib/python3.10/test/__pycache__/ |
Upload File : |
o
�9j�+ � @ s( d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlm Z d dlm
Z
d dl m Z m
Z
G dd� d�ZG dd � d eej�ZG d
d � d eej�ZG d d
� d
eej�ZG dd� deej�ZG dd� de�ZG dd� deej�ZG dd� deej�ZG dd� de�ZG dd� deej�ZG dd� deej�ZG d d � d eej�ZG d d � d eej�ZG d d!� d!e�Z G d"d#� d#eej�Z G d$d%� d%eej�Z G d&d'� d'eej�Z G d(d)� d)eej�Z G d*d+� d+eej�Z!G d,d-� d-eej�Z"G d.d/� d/e �Z#G d0d1� d1ej�Z$G d2d3� d3ej�Z%e&d4k�re�'� dS dS )5� N)�support)�
socket_helper)�threading_helper)�BaseHTTPRequestHandler�
HTTPServerc @ sH e Zd ZdZdZg Zg ZdZdd� Zdd� Z dd � Z
d
d � Z d d
� Z dS )�
BaseRobotTest� �test_robotparserNc C s, t �| j��� }tj�� | _| j�|� d S �N) �io�StringIO�
robots_txt� readlines�urllib� robotparser�RobotFileParser�parser�parse)�self�lines� r �,/usr/lib/python3.10/test/test_robotparser.py�setUp s zBaseRobotTest.setUpc C s$ t |t�r
|\}}||fS | j|fS r
)�
isinstance�tuple�agent�r �urlr r r r �get_agent_and_url s
z BaseRobotTest.get_agent_and_urlc C �` | j D ]*}| �|�\}}| j||d�� | �| j�||�� W d � n1 s(w Y qd S �N)r r )�goodr �subTest�
assertTruer � can_fetchr r r r �test_good_urls �
���z BaseRobotTest.test_good_urlsc C r r )�badr r" � assertFalser r$ r r r r �
test_bad_urls$ r& zBaseRobotTest.test_bad_urlsc C s | � | j�� | j� d S r
)� assertEqualr � site_maps�r r r r �test_site_maps* � z BaseRobotTest.test_site_maps)
�__name__�
__module__� __qualname__r
r r! r' r+ r r r% r) r- r r r r r s r c @ s e Zd ZdZddgZg d�ZdS )�UserAgentWildcardTestz�User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
�/�
/test.html)�/cyberworld/map/index.htmlz/tmp/xxx� /foo.htmlN�r/ r0 r1 r
r! r' r r r r r2 . s r2 c @ s e Zd ZdZg d�ZdgZdS )� CrawlDelayAndCustomAgentTestz�# robots.txt for http://www.example.com/
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
)r3 r4 )� cybermapperr5 r5 Nr7 r r r r r8 9 s
r8 c @ s& e Zd ZdZddgZdgZddgZdS )� SitemapTesta # robots.txt for http://www.example.com/
User-agent: *
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
r3 r4 r5 z7http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xmlz2http://www.google.com/hostednews/sitemap_index.xmlN)r/ r0 r1 r
r! r' r+ r r r r r: J s
�r: c @ s e Zd ZdZg Zg d�ZdS )�RejectAllRobotsTestz(# go away
User-agent: *
Disallow: /
)r5 r3 �/tmp/Nr7 r r r r r; [ s r; c @ s e Zd ZdZdZdd� ZdS )�BaseRequestRateTestNc C s� | j }| j| j D ]U}| �|�\}}| j||d��= | �|�|�| j� |�|�}| �|| j� | jd urO| �|t j
j � | �|j | jj � | �|j
| jj
� W d � n1 sYw Y q d S r )r r! r' r r" r* � crawl_delay� request_rate�assertIsInstancer r � RequestRate�requests�seconds)r r r r �parsed_request_rater r r �test_request_ratei s0
���� ���z%BaseRequestRateTest.test_request_rate)r/ r0 r1 r? r> rE r r r r r= e s r= c @ s e Zd ZdZdgZdS )�
EmptyFileTestr z/fooN)r/ r0 r1 r
r! r r r r rF � s
rF c @ s4 e Zd ZdZdZej�dd�ZdZ dgZ
g d�Z dS ) � CrawlDelayAndRequestRateTestz�User-agent: figtree
Crawl-delay: 3
Request-rate: 9/30
Disallow: /tmp
Disallow: /a%3cd.html
Disallow: /a%2fb.html
Disallow: /%7ejoe/index.html
�figtree� � � )rH r6 )�/tmpz /tmp.html� /tmp/a.html� /a%3cd.html� /a%3Cd.htmlz /a%2fb.htmlz/~joe/index.htmlN) r/ r0 r1 r
r r r rA r? r> r! r' r r r r rG � s rG c @ � e Zd ZdZdS )�DifferentAgentTestz FigTree Robot libwww-perl/5.04N�r/ r0 r1 r r r r r rQ � � rQ c @ s" e Zd ZdZdgZg d�ZdZdS )�InvalidRequestRateTestz�User-agent: *
Disallow: /tmp/
Disallow: /a%3Cd.html
Disallow: /a/b.html
Disallow: /%7ejoe/index.html
Crawl-delay: 3
Request-rate: 9/banana
rL )r< rM rN rO z /a/b.htmlz/%7Ejoe/index.htmlrK N)r/ r0 r1 r
r! r' r> r r r r rT � s
rT c @ s e Zd ZdZdgZg ZdS )�InvalidCrawlDelayTestz2User-Agent: *
Disallow: /.
Crawl-delay: pears
r6 Nr7 r r r r rU � s rU c @ � e Zd ZdZdZdgZdgZdS )� AnotherInvalidRequestRateTestzeUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Request-rate: whale/banana
� Googlebot�/folder1/myfile.html�/folder1/anotherfile.htmlN�r/ r0 r1 r
r r! r' r r r r rW � s
rW c @ s e Zd ZdZdZdgZdS )�UserAgentOrderingTestzMUser-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
rX z/something.jpgN)r/ r0 r1 r
r r' r r r r r\ � s
r\ c @ rP )�UserAgentGoogleMobileTestzGooglebot-MobileNrR r r r r r] � rS r] c @ rV )�GoogleURLOrderingTestzJUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
� googlebotrY rZ Nr[ r r r r r^ � s
r^ c @ � e Zd ZdZdgZdgZdS )�DisallowQueryStringTestz2User-agent: *
Disallow: /some/path?name=value
�
/some/pathz/some/path?name=valueNr7 r r r r ra � s
ra c @ r` )� UseFirstUserAgentWildcardTestzNUser-agent: *
Disallow: /some/path
User-agent: *
Disallow: /another/path
z
/another/pathrb Nr7 r r r r rc � s
rc c @ r` )�EmptyQueryStringTestz>User-agent: *
Allow: /some/path?
Disallow: /another/path?
z /some/path?z/another/path?Nr7 r r r r rd � s
rd c @ s0 e Zd ZdZej�dd�ZdZddgZ dgZ
dS ) �DefaultEntryTestzOUser-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/
rK � � r3 r4 r5 N) r/ r0 r1 r
r r rA r? r> r! r' r r r r re s
re c @ s e Zd ZdZdZdd� ZdS )�StringFormattingTestz�User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow: /some/path
zxUser-agent: cybermapper
Disallow: /some/path
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/c C s | � t| j�| j� d S r
)r* �strr �expected_outputr, r r r �test_string_formatting* r. z+StringFormattingTest.test_string_formattingN)r/ r0 r1 r
rj rk r r r r rh s
rh c @ s e Zd Zdd� Zdd� ZdS )� RobotHandlerc C s | � dd� d S )Ni� zForbidden access)�
send_errorr, r r r �do_GET0 s zRobotHandler.do_GETc G s d S r
r )r �format�argsr r r � log_message3 s zRobotHandler.log_messageN)r/ r0 r1 rn rq r r r r rl . s rl c @ s* e Zd Zdd� Zdd� Zejdd� �ZdS )� PasswordProtectedSiteTestCasec C sP | � tjj� ttjdft�| _t j
d| jj ddid�| _ d| j _
| j �� d S )Nr zHTTPServer serving�
poll_intervalg{�G�z�?)�name�target�kwargsT)�
addCleanupr �request�
urlcleanupr r �HOSTrl �server� threading�Thread�
serve_forever�t�daemon�startr, r r r r 9 s �z#PasswordProtectedSiteTestCase.setUpc C s" | j �� | j�� | j �� d S r
)r{ �shutdownr �join� server_closer, r r r �tearDownI s
z&PasswordProtectedSiteTestCase.tearDownc C s\ | j j}dtj d t|d � }|d }tj�� }|�|� |� � | �
|� d|�� d S )Nzhttp://�:rg z /robots.txt�*) r{ �server_addressr rz ri r r r �set_url�readr( r$ )r �addrr �
robots_urlr r r r �testPasswordProtectedSiteN s
z7PasswordProtectedSiteTestCase.testPasswordProtectedSiteN)r/ r0 r1 r r� r � reap_threadsr� r r r r rr 7 s
rr c @ sF e Zd ZdZd�e�Zedd� �Zdd� Zdd� Z d d
� Z
d d � Z d
S )�NetworkTestCasezhttp://www.pythontest.net/z{}elsewhere/robots.txtc C sV t �d� t�| j�� tj�| j�| _ | j �
� W d � d S 1 s$w Y d S )N�network) r �requiresr �transient_internet�base_urlr r r r
r r� )�clsr r r �
setUpClass^ s
"�zNetworkTestCase.setUpClassc C s& d� | j|tj�|�d sd�S d�S )Nz{}{}{}rg r3 r )ro r� �os�path�splitext)r r� r r r r e s
��zNetworkTestCase.urlc C sV | � | jj� | � | jj� | �| j�� d� | � | j�d�� | � | j�d�� d S )Nr r� )r( r � disallow_all� allow_all�
assertGreater�mtimer> r? r, r r r �
test_basicj s
zNetworkTestCase.test_basicc C s� | � | j�d| �d��� | �| j�d| j�� | �| j�d| �d��� | �| j�d| �d��� | �| j�d| �d��� | � | j�d| j�� d S )Nr� � elsewhere�Nutch�brian�webstats)r# r r$ r r( r� r, r r r �test_can_fetchq s z NetworkTestCase.test_can_fetchc C sf t j�| �d��}|�� | �|j� | �|j� | � |�
� d� | � |� d�� | � |�
d�� d S )Nz i-robot.txtr r� )r r r r r� r# r� r( r� r* r� � assertIsNoner> r? )r r r r r �
test_read_404y s z NetworkTestCase.test_read_404N) r/ r0 r1 r� ro r
� classmethodr� r r� r� r� r r r r r� Y s
r� �__main__)(r r� r| �unittest�urllib.robotparserr �testr � test.supportr r � http.serverr r r �TestCaser2 r8 r: r; r= rF rG rQ rT rU rW r\ r] r^ ra rc rd re rh rl rr r� r/ �mainr r r r �<module> sF "
"
) �