Go CrawlFarm Code October 5, 2016
I like to make an inventory of big software packages that I create, here it is for my crawlfarm.
I have added the code to Github
Please check it out. I will be making bug fixes and updating code to be more consistent and better. It's a pretty awesome project that I'm pretty proud of :)
Key Golang features, if you want to view it for example code:
- Gob
- Concurrent For / Select on Channels
- Logging using log package
- net.Conn, Server and Client, Dial and Accept
- sync.Mutex
- encoding/json
- time.NewTicker
- Synchronous and Asynchronous channels
- net/http.Client, CheckRedirect
- Regex to find href/src attributes
Feel free to fork and create pull requests :)