This is an exported version of the JIRA issue tracker. Please use the Google Code site to open new tickets or report updates to these existing tickets. Feel free to contact the mailing list with any questions.

[SCB-3] Needs to respect robots.txt before being unleashed fully
Created: Wed, 20 Aug 2003 21:11:57 -0700 (PDT)  Updated: Wed, 17 Jan 2007 13:08:45 -0800 (PST)

Status:Closed
Project:Scraping
Component/s:link-hunter
Affects Version/s:
Fix Version/s:scraping-engine-0.1

Type:New FeaturePriority: Major
Reporter:Henri YandellAssignee:Henri Yandell
Resolution:Fixed 
Environment:


 Description   
I need to have a plugin robots.txt understander.
Comment by bayard [ Wed, 20 Aug 2003 22:29:29 -0700 (PDT) ]
org.osjava.scraping.norobots.NoRobotClient is begun as a part of the scraping-engine.

One problem is that it only needs to be used on HTTP [and HTTPS?] traffic, so it's linked to the protocol. Possibly it would be a configuration option per protocol?
Comment by bayard [ Thu, 21 Aug 2003 21:27:22 -0700 (PDT) ]
Robots.txt handler written. Needs plugging in.
Comment by bayard [ Fri, 22 Aug 2003 09:38:44 -0700 (PDT) ]
This is now a project called Norbert. So need to change Scabies to use Norbert.
Comment by bayard [ Tue, 11 Nov 2003 22:25:27 -0800 (PST) ]
The engine now pays attention to robots.txt unless overridden in the configuration.
This is applied only to http/https and uses norbert.
Comment by bayard [ Tue, 11 Nov 2003 22:29:33 -0800 (PST) ]
Tested on slashdot's robots.txt fiel with success.
Comment by cisco77 [ Wed, 17 Jan 2007 13:08:44 -0800 (PST) ]

http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-201.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-200.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-199.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-198.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-197.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-196.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-195.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-194.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-193.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-192.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-191.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-190.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-189.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-188.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-187.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-186.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-185.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-184.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-183.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-182.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-181.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-180.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-179.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-178.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-177.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-176.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-175.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-174.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-173.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-172.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-171.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-170.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-169.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-168.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-167.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-166.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-165.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-164.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-163.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-162.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-161.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-160.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-159.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-158.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-157.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-156.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-155.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-154.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-153.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-152.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-151.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-150.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-149.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-148.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-147.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-146.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-145.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-144.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-143.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-142.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-141.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-140.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-139.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-138.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-137.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-136.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-135.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-134.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-133.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-132.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-131.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-130.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-129.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-128.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-127.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-126.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-125.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-124.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-123.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-122.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-121.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-120.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-119.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-118.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-117.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-116.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-115.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-114.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-113.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-112.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-111.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-110.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-109.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-108.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-107.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-106.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-105.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-104.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-103.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-102.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-101.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-100.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-99.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-98.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-97.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-96.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-95.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-94.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-93.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-92.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-91.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-90.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-89.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-88.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-87.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-86.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-85.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-84.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-83.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-82.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-81.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-80.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-79.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-78.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-77.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-76.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-75.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-74.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-73.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-72.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-71.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-70.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-69.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-68.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-67.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-66.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-65.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-64.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-63.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-62.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-61.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-60.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-59.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-58.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-57.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-56.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-55.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-54.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-53.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-52.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-51.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-50.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-49.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-48.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-47.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-46.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-45.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-44.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-43.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-42.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-41.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-40.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-39.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-38.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-37.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-36.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-35.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-34.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-33.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-32.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-31.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-30.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-29.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-28.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-27.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-26.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-25.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-24.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-23.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-22.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-21.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-20.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-19.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-18.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-17.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-16.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-15.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-14.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-13.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-12.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-11.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-10.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-9.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-8.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-7.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-6.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-5.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-4.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-3.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-2.html
http://cisco21.onlyphpbb.de/board/nxtopicf35446363nx7776-1.html