Nadim Tuhin
Published on

DSE Scrapper: A Journey into Bangladesh Stock Market Data

stock-marketphpscrapingprojectopen-source
Authors

Back in 2015, during the peak of my PHP development days, I embarked on an ambitious journey to create a comprehensive stock market platform. The DSE Scrapper project was born out of my enthusiasm for the stock market and a desire to make market data more accessible to Bangladeshi investors.

The Vision

The project started with a grand vision: to build a complete stock market platform featuring:

  • Real-time stock listings
  • Portfolio management
  • Market analysis tools
  • Historical data tracking

As a PHP developer heavily influenced by Laravel patterns, I approached this project with the intention of creating a robust and maintainable codebase.

Technical Challenges

The primary challenge was accessing data from the Dhaka Stock Exchange. Web scraping seemed like the obvious solution, but it came with its own set of obstacles:

  • Access Restrictions: DSE's website initially blocked scraping attempts
  • Data Structure: The need to parse and structure the scraped data effectively
  • Real-time Updates: Handling frequent data updates without overwhelming the server

The Breakthrough

The key breakthrough came when I discovered that adding a referrer header to the cURL requests would allow successful scraping. This simple yet crucial finding made the entire project possible:

curl_setopt($ch, CURLOPT_REFERER, 'http://www.dse.com.bd');

Learning Experience

While the project didn't reach its full potential due to work commitments and time constraints, it provided valuable lessons:

  • Technical Skills: Deep dive into web scraping techniques and data processing
  • Project Management: Understanding the importance of scope management
  • Persistence: The value of solving technical challenges step by step

Code Architecture

The project followed Laravel-inspired patterns:

  • Clean separation of concerns
  • Service-based architecture
  • Robust error handling
  • Configurable scraping parameters

Impact and Legacy

Though the project remained incomplete, it served as:

  • A learning platform for understanding stock market data structures
  • A template for future web scraping projects
  • A reminder that even unfinished projects contribute to our growth as developers

Conclusion

The DSE Scrapper project, while not reaching its ambitious goals, represents an important chapter in my development journey. It showcases how enthusiasm for a domain (stock market) combined with technical skills can lead to interesting projects, and how even "incomplete" projects can provide valuable learning experiences.

The project remains open-source and available on GitHub for anyone interested in building upon it or learning from its approach to web scraping.

Discuss on TwitterView on GitHub