Member Of Technical Staff (Web Access Engineer, Crawler)
perplexity
Job Score
100 ptsPerplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems.
Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.
Responsibilities
Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.
Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.
Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.
Qualifications
5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.
Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.
Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.
Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.
Comfort working independently when the path forward is not fully defined.
Nice to have
Experience developing CDN, caching, or web performance systems.
Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.
Contributions to a browser engine or JavaScript runtime.
About Office Suite
Proficiency in Office Suite is an essential skill for professionals across various areas. Microsoft Excel, Word, PowerPoint, and Outlook are fundamental tools in the corporate day-to-day, while Google Workspace and other collaborative solutions are gaining increasing space.
Key skills include advanced Excel (formulas, pivot tables, Power Query, VBA/Macros), Word (formatting, mail merge, styles), PowerPoint (presentation design, animations), Outlook (email and calendar management), and collaborative tools like Google Sheets, Notion, and Airtable.
Professionals with advanced Office Suite proficiency are valued in administrative, financial, data, and operational areas. Microsoft Office Specialist (MOS) certifications are important differentiators. The combination of advanced Excel with data analysis skills is one of the most in-demand competencies in the market.
About Account Manager
The Account Manager is the professional responsible for managing and expanding the relationship with clients after the sale. They act as a strategic partner, ensuring satisfaction, retention, and account growth, connecting client needs with company solutions.
Key skills include relationship management, negotiation, upsell and cross-sell, contract renewal, account planning, business reviews, metrics analysis (NPS, churn, LTV), and CRM knowledge (Salesforce, HubSpot). Communication, empathy, and business vision are fundamental differentiators.
Account Managers in technology and SaaS companies are highly valued, especially those who can increase recurring revenue (MRR/ARR) through account expansion and churn prevention. The field offers opportunities from account executive to director of accounts, with a focus on strategic relationship, revenue growth, and customer success.
Discover Other Areas
Understand the scope of work, key skills, and tools used in different career areas.
About Human Resources
The Human Resources area is responsible for all people management in organizations, from attracting and selecting talent to developing, retaining, and ensuring employee well-being. HR professionals are fundamental to building strong organizational cultures and engagement.
Key skills include recruitment and selection, compensation and benefits management, learning and development (L&D), organizational climate, employee engagement, labor law, labor relations, and HR tools (Workday, SAP SuccessFactors, Bamboo HR). Knowledge of people analytics and data-driven HR is a differentiator.
HR professionals in technology companies are highly valued, especially those who master employer branding, people analytics, and talent retention strategies. The field offers opportunities from HR analyst to Chief People Officer, with a focus on culture, engagement, and people growth.
About Copywriting
The Copywriting area is responsible for creating persuasive, creative, and strategic texts for various communication channels. Copywriting professionals transform ideas into words that engage, convert, and build brand voice.
Key skills include advertising copywriting, script writing for videos and podcasts, persuasive writing, tone of voice, and editorial guidelines. Knowledge of SEO writing, Grammarly, and text productivity tools is a differentiator.
Copywriting professionals in technology companies are highly valued, especially those who master copy for landing pages, email sequences, and funnel content. The field offers opportunities from junior copywriter to head of copy, with a focus on creativity, persuasion, and performance.
About Agile
The Agile and Digital Transformation area is fundamental for organizations seeking efficiency and rapid adaptation. Agile professionals facilitate processes, eliminate bottlenecks, and promote a culture of continuous improvement.
Key certifications include CSM, PSM, SAFe, ICP, and Kanban. Knowledge of Scrum, Kanban, XP, and agile frameworks is essential, as are leadership and facilitation soft skills.
Senior Agile coaches and Scrum Masters are highly valued, especially in technology companies that adopt agile methodologies at scale.
About Sales
The Sales area is responsible for generating revenue and expanding the customer base. B2B and B2C sales professionals are fundamental for sustainable growth of any organization.
Key skills include prospecting, negotiation, CRM (Salesforce, HubSpot), sales enablement, and value consulting. The consultative and data-driven approach is increasingly valued.
Consultative sellers and senior Sales Managers have very high earning potential, with OTE (On-Target Earnings) that can exceed monthly salaries in technology companies.
About Customer Service
The Customer Service / Client Relations area is essential for ensuring customer satisfaction, retention, and a good relationship. Professionals in this field are the primary interface for communication, handling inquiries, requests, feedback, and ensuring a high-quality day-to-day experience. Skills in communication, problem-solving, empathy, and patience are indispensable.
Comments 0