> For the complete documentation index, see [llms.txt](https://dev-angelist.gitbook.io/ewptv2-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dev-angelist.gitbook.io/ewptv2-notes/readme/network-security/2.2-passive-crawling-and-spidering.md).

# 2.2 Passive Crawling & Spidering

**Crawling** and S**pidering** are techniques used to automatically browse and index web content, typically performed by search engines or web crawlers. Here's a breakdown of each:

## **Crawling**

* Crawling is the process of systematically browsing the web to discover and index web pages.
* A crawler, also known as a web spider or web robot, starts with a list of seed URLs and then follows hyperlinks from one page to another, recursively.
* Crawlers retrieve web pages, extract links from them, and add new URLs to their queue for further crawling.
* They often adhere to a set of rules specified in a file called robots.txt to determine which parts of a website they are allowed to crawl and index.
* We can utilize [**Burp Suite**](https://dev-angelist.gitbook.io/practical-ethical-hacker-ceh-tools/practical-ethical-hacker-notes/tools/burp-suite)'s passive crawler to map out the web app and understand its structure (navigating on website an retrieve the site map).

## **Spidering**

* Spidering is a term often used interchangeably with crawling, referring to the automated process of fetching web pages and following links.
* It derives from the analogy of a spider weaving a web by moving from one location to another and creating connections.
* Spidering involves systematically traversing the web, exploring web pages and their links to index content for search engines or other purposes.
* Spidering algorithms may vary based on the specific objectives, such as optimizing for depth-first or breadth-first traversal, or prioritizing certain types of content.
* We can utilize Burp Suite Pro or [**OWASP ZAP**](https://dev-angelist.gitbook.io/practical-ethical-hacker-ceh-tools/practical-ethical-hacker-notes/tools/zap)'s Spider to automate the process of spidering a web application to map out the web app and understand its structure.

### Spidering with OWASP ZAP

* Abilitate Foxy Proxy on browser (Firefox)
* Refresh webpage to retrieve Sites Link on ZAP
* Click on Tools -> Spider
* Select URL and abilitate recurse
* Start Scan

<figure><img src="https://1357648772-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FesKkkNq5JVqzRLKWf4Ug%2Fuploads%2FjkBZNsBgRLOqKmcD4Bfq%2Fimage.png?alt=media&amp;token=9c60a7a0-8b62-4248-bc89-e5e895c50726" alt=""><figcaption></figcaption></figure>

We obtained more URIs and we can save them into .csv format for a better exportation and manipulation.

<figure><img src="https://1357648772-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FesKkkNq5JVqzRLKWf4Ug%2Fuploads%2FF2NWej1j4mi18PPogDtq%2Fimage.png?alt=media&amp;token=415b3a8d-b632-466d-8767-88352724b602" alt=""><figcaption></figcaption></figure>