Observability Engineer

  • BlockFi
  • Remote (New York, NY, USA)
  • Nov 18, 2021
Full time Security & Privacy

Job Description

What is BlockFi?

BlockFi’s mission is to provide liquidity, transparency and efficiency to digital financial markets by creating products that meet the needs of consumers and corporations across the globe. We build bridges between traditional finance and digital markets that enable growth for all participants.

We’re a team of builders and strivers, proud to champion financial inclusivity and offer economic opportunities around the globe. We provide the same inclusivity to our team members. BlockFi is a place where diversity is celebrated, individuality is recognized, and every single team member is valued. We are changing the status quo to be the first financial company that people love, and we rely on our people to make it happen!

BlockFi is looking for an Observability Engineer to join our growing team! 

About Your Team

The Observability team works remotely across different timezones and regions. They help the product and engineering organizations  explore and understand our most critical systems. We must find the balance of sorting through the noise to find the signal using deep knowledge of our platform. Our team's mission is to provide development environments, tools, and infrastructure to enable BlockFi engineers to build product features and shepherd them to production with confidence.

Your Mission

As an Observability Engineer, you will build tools to measure the availability and performance of the BlockFi platform. Our work highly varies in its nature, ranging from writing and instrumenting services in Elixir and Python, to deploying infrastructure, mentoring junior engineers, giving talks and demos, and generally promoting best practices throughout the organization.


  • Implement and maintain observability and monitoring tools across the entire stack; logging systems & aggregators, metrics monitoring tools, distributed tracing platforms, etc. 
  • Automate system processes to increase our velocity in managing uptime and risk mitigation
  • Manage monitoring systems to support customer metrics delivery dashboards 
  • Debug issues and solve distributed systems challenges across services and levels of the stack
  • Educate software & infrastructure engineering teams on observability & monitoring best practices
  • Drive the continued evolution of monitoring tooling towards a standards-based self-service automated platform
  • Engage with product teams on monitoring and observability implementations while fostering a culture of reliability with an SLO-based approach

Your Expertise 

  • Experience: 5+ years in a DevOps, Infrastructure, Site Reliability,  or similar role.
  • Depth in Observability: A deep grasp of the value of Observability as a practice is instrumental.
  • Debugging:  You can debug both human and automated processes. You don't mind touching multiple languages (We use Elixir, Python, Rust primarily) and have a good understanding of how distributed systems interact and what their failure points are.
  • Exposure and expertise with
    • Logging systems and concepts, such as ELK, Splunk, and Grafana Loki
    • Metrics monitoring systems, such as Grafana, Prometheus, InfluxDB, etc...
    • Distributed tracing platforms such as Jaeger, Zipkin, Honeycomb, etc...
  • Communication: Strong written and spoken skill, we are an international company so being able to clearly share and document your thoughts and work is critical to your’s and the team's success
  • Bonus Points for
    • Prior experience writing and maintaining Erlang/Elixir based systems.
    • GRC: Knowledge and experience Governance, Risk, and Compliance best practices as it concerns regulations such as GDPR, CCPA, and other data protection regulations.
    • Enjoy teaching: Educating the product team on tool usage and instrumentation fosters a resilient team. So you don't mind dropping what you're working on from time to time to help an engineer from a different team instrument their code or debug something. You know how to build a compelling story and look forward to giving talks and demos.

Your Perks:

We benefit from the great work our employees do each day. That is why we are committed to providing a variety of awesome benefits to help them live their best lives.

  • Competitive salary because we value your experience and expertise
  • Unlimited vacation / sick days because everyone deserves time for R&R
  • Employer contribution towards health coverage (including vision & dental) because your physical health and well-being is important to us 
  • Various fringe benefits such as 401k, Parental Leave, FSA/HSA, and Employee Assistance Programs because health coverage is more than just choosing your yearly plan 
  • Flexible work environment because we are a geographically dispersed team and we believe in balance
  • Pet insurance because all of your beloved family members should have coverage too
  • Weekly lunch stipend because there is such a thing as a free lunch! 
  • A close-knit team of enthusiastic, collegial and driven people to work alongside in a highly meritocratic environment because teamwork makes the dreamwork

Why BlockFi?

BlockFi has experienced incredible growth since our launch in August 2017. Our client base has grown to more than 225,000 (and counting), and the company now boasts more than $15 billion in assets on our platform. We recently completed a Series D funding round placing the company's valuation at $3 billion, and our team now has more than 500 people worldwide. We have established ourselves as a crypto market leader, and as we expand our product suite and geographic footprint, we expect our addressable market to grow exponentially.

BlockFi's leadership team has decades of experience in the traditional financial services and banking world, and we take a conservative approach to regulation that will position us well for sustainable long-term growth and expansion.

Our team is comprised of highly motivated professionals from diverse backgrounds. We are aiming to become the leading lender in crypto and are poised to redefine the global financial ecosystem for the better. In addition:

  • BlockFi is one of the first companies to ever offer crypto-backed loans and the only company whose founding team has an institutional understanding of the debt capital markets and regulatory landscape in the U.S.
  • $100 MM of Series A, B, and C funding led by Valar Ventures with participation from Susquehanna, Winklevoss Capital, Fidelity, Galaxy Digital, Akuna Capital, and Morgan Creek
  • $350 MM of Series D funding led by Bain Capital Ventures, partners of DST Global, Pomp Investments and Tiger Global
  • We are moving quickly and have already deployed substantial capital into the space, proving our ability to execute and capture customer demand

For information about how we collect, use, or otherwise handle your Personal Information, please see our Privacy Policy.