Alright, so I’ve been messing around with this “gau” thing, trying to get all the URLs and endpoints from websites. It’s kinda like a treasure hunt, but instead of gold, you’re finding, like, secret doors on a website.
First, I installed it. That was kinda a pain, honestly. I had to get Go installed first, which was a whole other adventure. I’m not gonna lie, I spent, like, an hour just figuring out how to get Go working properly. I followed some online * seemed to work.
Once I had Go set up, installing gau was supposed to be easy. You just type in this command:
go install */lc/gau/v2/cmd/gau@latestBut, of course, it wasn’t that easy for me. It threw some errors, I had to fiddle with some settings(I had no idea I did before.), and finally, it seemed to install okay. I honestly don’t even remember what I did, I just kept Googling stuff until it worked.
Then, the fun part (sort of). I picked a website I wanted to poke around on. Let’s call it “*” just to keep things simple. To actually use gau, I typed this into my terminal:
gau *And…bam! A whole bunch of stuff started scrolling by. Like, a LOT of stuff. URLs, endpoints, Javascript files, you name it. It was kinda overwhelming, to be honest. It felt like drinking from a firehose.
So, I learned you can pipe the output to a file, which makes it way easier to deal with. I did this:
gau * > example_*That created a text file called “example_*” with all the results. Much better! Now I could actually look through it without my eyes glazing over.
I opened the file, and started scrolling. I saw some interesting stuff, some stuff I didn’t understand, and some stuff that probably shouldn’t be public. That’s the wild west of the internet for you, I guess.
Now for the “all american” part. It is a parameter to get more comprehensive results, so I tried this:
gau --providers wayback,otx,commoncrawl *
This means, I tried to collect data from multiple sources to get more endpoints. I outputted the data I got from here into another txt file, and compare with the previous results.
To be perfectly honest, I’m still figuring out what half of this stuff even means. I’m just clicking around, trying things out, and seeing what happens. But it’s pretty cool to see all the hidden parts of a website that you normally don’t get to see. It’s like having X-ray vision for the internet.
It also has some other features which I haven’t got chance to try them * I think I am gonna use it more often.
