openflarestack Star this repo
  1. Catalog
  2. Developer tools
  3. Deepcrawl
Ranked 52 of 234

Deepcrawl

lumpinif/deepcrawl

Turns web pages into clean markdown and link trees for AI agents, with a dashboard.

A self-hosted alternative to Firecrawl.

Install with CLIView source

What it sets up in your account

  • KV
    KV namespace
    A key value store for settings and fast lookups.
  • D1
    D1 database
    A SQL database for the app’s records.
Category
Developer tools
Licence
MIT
Stars
688
Last commit
7 weeks ago
Latest release
deepcrawl@0.5.5

Use it with AI or code

  • Markdown for AIPaste into ChatGPT, Claude or any LLM./apps/deepcrawl.md
    Open
  • JSON APIFree, no key, CORS open./api/v1/apps/deepcrawl.json
    Open

More in the API docs and llms.txt.

Keeping it up to date

Run this in your copy whenever a new version comes out.

terminal
git remote add upstream https://github.com/lumpinif/deepcrawl.git
git pull upstream main
npx wrangler deploy

Deepcrawl questions

What is Deepcrawl?

Turns web pages into clean markdown and link trees for AI agents, with a dashboard. It’s an open-source, self-hosted alternative to Firecrawl. It runs on Cloudflare Workers in your own account.

What does Deepcrawl set up in my Cloudflare account?

KV namespace and D1 database. They are created in your account, so the data stays yours.

How do I deploy Deepcrawl?

It installs from the command line. Follow the steps in the lumpinif/deepcrawl README.

Is Deepcrawl free and open source?

Yes. It’s published under the MIT licence. Running it uses your own Cloudflare account, and small setups usually fit in the free plan.

How do I update Deepcrawl?

Pull from the original repository and redeploy: add https://github.com/lumpinif/deepcrawl.git as an upstream remote, run git pull upstream main, then npx wrangler deploy.