Executing advanced background queries constantly expands context windows sequentially until the system reaches strict absolute limits (~200,000 constraint markers) resulting natively in prompt too long (413 HTTP) rejections. Claude evades context bottlenecks natively using sequential Compaction.
Token accuracy equates explicitly to memory survival. Rather than calculating via strict expensive API counting payloads perpetually, the local state mixes API counts natively interspersed with highly optimized heuristics roughly mapping (~1 raw token sequentially per 4 plain string characters).
When the loop runs natively, it explicitly reduces token footprint across 5 progressive limits:
20,000 max string limits returning .md formatted file storage indicators alternatively.HISTORY_SNIP blocks natively reducing overhead maintaining exclusively the formal execution outline parameters natively.cache_edits payloads to silently clear transient backend tool memory configurations directly piggybacking adjacent network transfers.When total sizes breach the internal warning bounds (~13k overhead), Claude activates an asynchronous operation natively entirely executing independent summarization branches silently.
Session Memory limits are built structurally extracting exact project paths (like logging # Errors & Corrections) dumping entirely via hidden memory paths locally mapped directly inside .claude/session_memory.
The central prompt engine purges everything before the boundary checkpoint retaining fully logically mapped histories! The terminal executes cleanly utilizing infinitely looping logic frameworks natively extending context survival globally.
We hope this 3-part series expanded your perspective massively surrounding modern LLM backend constraints!
]]>When scaling agentic LLMs against massive enterprise directories containing thousands of components, utilizing native JavaScript filesystem commands is prohibitively slow. Claude rectifies this explicitly by offloading operations directly to local system rust binaries.
GrepTool natively executes Ripgrep mappings by actively bypassing node-bound file reads. To ensure zero cold-start deployment delays, Claude bundles embedded binaries securely avoiding $PATH environment resolution blocking delays.
One of the cleverest performance safeguards relates to thread saturation natively.
Inside constrained systems (like Docker execution limits), initiating overly aggressive rg parallel thread pools destroys resources, triggering an explicit EAGAIN (resource temporarily unavailable) exception.
Claude intercepts the exact EAGAIN network signature dynamically. Rather than entirely terminating the extraction loop, it scales the process silently downwards, injecting the -j 1 argument (single-threaded execution constraint). It trades overall execution speed exclusively for guaranteed terminal robustness.
When analyzing search pipelines, returning 20,000 matches absolutely destroys token budgets instantly. How does Claude search without overflowing context models?
The entire search parameter uses hard-coded output buffers (a static head_limit). Should Ripgrep find thousands of elements, it truncates the payload severely offering only exact samples wrapped by the system indication:
[Showing results with pagination = limit: 250]
The LLM is programmed internally to recognize optimization block limits logically prompting an incremental offset: 250 execution step recursively. Searching your entire repository essentially behaves exactly like iterating over dynamic relational database buffers via cursor endpoints natively.
In Part 3, we will move into Claude’s greatest performance implementation yet: The 5-layer Context Compaction parameters.
]]>For AI performance enthusiasts, seeing how an LLM agent wraps around a native Node.js interface seamlessly is exhilarating. Let’s unbox the core architectural structure behind the blindingly fast experience it delivers in the terminal.
The typical Node.js CLI tool crashes heavily on import cascades. Claude solves startup latency explicitly with heavily engineered Fast-Path Routing inside src/entrypoints.
Before any dependencies or large React/Ink components load, a trivial scan processes simple arguments. A static TCP connection dynamically initiates the Anthropic API ping parallelizing DNS resolutions before the UI even renders. Background promises concurrently establish configuration reads without blocking visual shell indicators.
When analyzing the execution layer (QueryEngine.ts), the primary focus is how it mitigates response latency. Total latency is completely eradicated by streaming chunks directly using for await. As Anthropic processes prompt batches natively, Claude pipes the raw byte yields instantaneously into the virtual UI render state.
To further lower API friction, Claude intercepts mutating terminal actions natively via local Zod validation schemas. Instead of waiting for a network error round-trip specifying schema failures, local Zod implementations block malformed JSON outputs generated by the LLM seamlessly forcing internal reroutes.
Performance doesn’t stop randomly at execution loops. In Part 2, we will address the scalability optimizations within Claude’s native Agentic Search engine, illustrating how it safely handles executing terminal binaries across localized directories infinitely.
To successfully extract data at scale without triggering CAPTCHA loops or IP bans, you must meticulously manage your network identity and decouple your scraping infrastructure.
The first line of defense for enterprise anti-bot solutions is IP reputation. Datacenter IP ranges (AWS, GCP, DigitalOcean) are universally blacklisted or aggressively rate-limited.
Once you have established a trusted network identity, the focus shifts to hardware utilization and execution design.
High-performance scraping must respect the ecosystem:
robots.txt unless you possess a high-value, legally sound use case for bypassing it.By combining high-quality residential networks with a decoupled, containerized architecture, you can build data pipelines capable of reliably handling massive throughput against modern enterprise defenses. If you’d rather skip the infrastructure setup, our ready-to-use scraping tools for TikTok, Twitter, and more already handle proxy management, browser orchestration, and anti-bot evasion at scale.
]]>For AI startups, this leak effectively commoditized Anthropic’s specific orchestration logic. Competitors across the ecosystem now have a detailed blueprint for complex LLM deployment:
Predictably, the leak triggered an immediate surge in malicious activity. Threat actors quickly published “cracked” or “unlocked” versions of Claude Code on GitHub and npm.
Many of these unofficial forks contain obfuscated malware—specifically Remote Access Trojans (RATs)—designed to systematically exfiltrate developers’ .env files and AWS credentials during “agentic” runs.
Technical Advisory: Engineering teams should audit local environments and instantly block unauthorized npm scopes. Running leaked or unofficial AI agent binaries presently poses a Tier-1 risk to corporate intellectual property and security.
This event serves as a brutal case study for Artifact Scanning in the CI/CD pipeline.
.map or .ts files are present in production distribution folders.The leak has sparked an intense, industry-wide debate regarding AI Transparency vs. Intellectual Property. While the foundation model Weights remain entirely secure behind Anthropic’s private API, the unprecedented exposure of the “System Instructions” (Prompts) highlights the vanishing line between a commercial product and its configuration.
Moving forward, the industry is highly likely to see an accelerated shift toward Model Context Protocol (MCP) standardization. As Anthropic’s leaked code clearly demonstrated, maintaining proprietary, highly-fragmented tool-calling logic is evolving from a competitive advantage into a severe maintenance burden and security liability. In the data extraction space, this same principle holds — standardized, well-audited scraping tools are replacing fragile bespoke scripts, lowering supply-chain risks while improving reliability.
The overarching takeaway is clear: In the age of AI, the “Agentic Logic”—how an AI thinks and uses tools—is just as valuable as the neural model underlying it.
]]>Series Navigation
- Part 1: Searching TikTok with the Apify Python Client
- Part 2: Downloading & Organizing Video Files
- Part 3: Auto-Generating News Scripts with AI
- Part 4: Scheduling & Publishing to YouTube ← You are here
client_secret.json)pip install google-auth google-auth-oauthlib google-api-python-client
Create youtube_auth.py:
# youtube_auth.py
import os
import pickle
from pathlib import Path
from google.auth.transport.requests import Request
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
SCOPES = ["https://www.googleapis.com/auth/youtube.upload"]
TOKEN_PATH = Path("credentials/youtube_token.pickle")
CLIENT_SECRET_PATH = Path("credentials/client_secret.json")
def get_authenticated_service():
"""Authenticate and return a YouTube API service object."""
credentials = None
if TOKEN_PATH.exists():
with open(TOKEN_PATH, "rb") as token:
credentials = pickle.load(token)
if not credentials or not credentials.valid:
if credentials and credentials.expired and credentials.refresh_token:
credentials.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
str(CLIENT_SECRET_PATH), SCOPES
)
credentials = flow.run_local_server(port=8090)
TOKEN_PATH.parent.mkdir(parents=True, exist_ok=True)
with open(TOKEN_PATH, "wb") as token:
pickle.dump(credentials, token)
return build("youtube", "v3", credentials=credentials)
The first time you run this, a browser window will open for Google OAuth consent. After that, the token is cached locally.
Create upload_youtube.py:
# upload_youtube.py
import sys
from datetime import datetime
from pathlib import Path
from googleapiclient.http import MediaFileUpload
from youtube_auth import get_authenticated_service
OUTPUT_DIR = Path("data/output")
def generate_metadata(keyword: str) -> dict:
"""Generate title, description, and tags for a YouTube upload."""
topic = keyword.replace("_", " ").title()
date_str = datetime.now().strftime("%B %d, %Y")
return {
"title": f"🔴 {topic} — Latest Updates | {date_str}",
"description": (
f"Today's coverage of {topic}. "
f"This segment compiles the latest on-the-ground footage "
f"and verified reports.\n\n"
f"⚠️ Some footage may be graphic. Viewer discretion advised.\n\n"
f"📌 Subscribe for daily global news updates.\n"
f"🔔 Turn on notifications to never miss breaking news.\n\n"
f"#GlobalNews #{keyword.replace('_', '')} #BreakingNews"
),
"tags": [
topic, "breaking news", "global news",
keyword.replace("_", " "), "world news",
"current events", "news today",
],
}
def upload_video(
youtube,
video_path: str,
title: str,
description: str,
tags: list[str],
category_id: str = "25", # News & Politics
privacy: str = "private",
):
"""Upload a single video to YouTube."""
body = {
"snippet": {
"title": title,
"description": description,
"tags": tags,
"categoryId": category_id,
},
"status": {
"privacyStatus": privacy,
"selfDeclaredMadeForKids": False,
},
}
media = MediaFileUpload(
video_path,
mimetype="video/mp4",
resumable=True,
chunksize=1024 * 1024 * 10, # 10 MB chunks
)
request = youtube.videos().insert(
part="snippet,status",
body=body,
media_body=media,
)
response = None
while response is None:
status, response = request.next_chunk()
if status:
pct = int(status.progress() * 100)
print(f" 📤 Uploading... {pct}%")
video_id = response["id"]
print(f" ✅ Uploaded: https://youtu.be/{video_id}")
return video_id
def main():
youtube = get_authenticated_service()
video_files = sorted(OUTPUT_DIR.glob("*_news_segment.mp4"))
if not video_files:
print("❌ No compiled segments found in data/output/")
sys.exit(1)
print(f"📋 Found {len(video_files)} segment(s) to upload.\n")
for video_path in video_files:
keyword = video_path.stem.replace("_news_segment", "")
meta = generate_metadata(keyword)
print(f"🎬 Uploading: {meta['title']}")
upload_video(
youtube,
str(video_path),
title=meta["title"],
description=meta["description"],
tags=meta["tags"],
privacy="private", # Change to "public" when ready
)
print()
print("🎉 All uploads complete!")
if __name__ == "__main__":
main()
Tip: Start with
privacy="private"to review your uploads before making them public.
Create pipeline.py to orchestrate the entire workflow in a single command:
# pipeline.py
import subprocess
import sys
from pathlib import Path
def run_step(script: str, args: list[str] = None):
"""Run a pipeline step and exit on failure."""
cmd = [sys.executable, script] + (args or [])
print(f"\n{'='*60}")
print(f"▶️ Running: {' '.join(cmd)}")
print(f"{'='*60}\n")
result = subprocess.run(cmd)
if result.returncode != 0:
print(f"\n❌ Step failed: {script}")
sys.exit(1)
def find_latest_results() -> str:
"""Find the most recent search results JSON file."""
results = sorted(Path("data/raw").glob("tiktok_results_*.json"))
if not results:
raise FileNotFoundError("No results found in data/raw/")
return str(results[-1])
def main():
print("🚀 Starting TikTok → YouTube Pipeline\n")
# Step 1: Search TikTok
run_step("search_tiktok.py")
# Step 2: Download videos
latest_json = find_latest_results()
run_step("download_videos.py", [latest_json])
# Step 3: Generate scripts
run_step("generate_script.py")
# Step 4: Generate TTS audio
run_step("generate_tts.py")
# Step 5: Compile videos
run_step("compile_video.py")
# Step 6: Upload to YouTube
run_step("upload_youtube.py")
print("\n🎉 Pipeline complete! Check your YouTube Studio.")
if __name__ == "__main__":
main()
Run the entire pipeline:
python pipeline.py
To run this pipeline automatically, you have two options:
For a simpler setup, add a cron job on your server:
# Run every 12 hours at 6:00 AM and 6:00 PM
0 6,18 * * * cd /path/to/project && /usr/bin/python3 pipeline.py >> logs/pipeline.log 2>&1
tiktok-youtube-pipeline/
├── .env
├── config.py
├── search_tiktok.py # Part 1
├── download_videos.py # Part 2
├── generate_script.py # Part 3
├── generate_tts.py # Part 3
├── compile_video.py # Part 3
├── youtube_auth.py # Part 4
├── upload_youtube.py # Part 4
├── pipeline.py # Orchestrator
├── credentials/
│ ├── client_secret.json
│ └── youtube_token.pickle
├── data/
│ ├── raw/ # JSON search results
│ ├── videos/ # Downloaded .mp4 files
│ ├── scripts/ # AI-generated narration
│ ├── audio/ # TTS audio files
│ └── output/ # Final compiled segments
└── logs/
Over this 4-part series, we built a complete automated pipeline that:
This pipeline transforms raw social media footage into a professional news channel — all with Python and a handful of APIs. The total cost per run is approximately $0.05 (Apify compute + OpenAI tokens), making it incredibly cost-effective for daily news production.
]]>Series Navigation
- Part 1: Searching TikTok with the Apify Python Client
- Part 2: Downloading & Organizing Video Files
- Part 3: Auto-Generating News Scripts with AI
- Part 4: Scheduling & Publishing to YouTube ← You are here
Series Navigation
- Part 1: Searching TikTok with the Apify Python Client
- Part 2: Downloading & Organizing Video Files
- Part 3: Auto-Generating News Scripts with AI ← You are here
- Part 4: Scheduling & Publishing to YouTube
brew install ffmpeg on macOS)pip install openai ffmpeg-python
Add your OpenAI key to .env:
OPENAI_API_KEY=sk-your_key_here
The TikTok video descriptions (desc field) captured in Part 1 contain valuable context — hashtags, locations, and brief captions. We’ll feed a batch of these to an LLM to produce a professional news script.
Create generate_script.py:
# generate_script.py
import json
import os
from pathlib import Path
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
METADATA_PATH = Path("data/videos/metadata_index.json")
SCRIPTS_DIR = Path("data/scripts")
SYSTEM_PROMPT = """You are a professional news anchor script writer.
Given a list of TikTok video descriptions about a specific topic,
write a concise, factual news narration script (60-90 seconds when
read aloud). The script should:
- Start with a strong headline opener
- Summarize the key events shown in the videos
- Maintain a neutral, journalistic tone
- End with a brief outlook or call to stay updated
Do NOT mention TikTok or social media in the script.
Output ONLY the script text, no titles or formatting."""
def load_metadata() -> dict[str, list[dict]]:
"""Load metadata and group by keyword."""
with open(METADATA_PATH, "r", encoding="utf-8") as f:
metadata = json.load(f)
grouped = {}
for item in metadata:
keyword = item["keyword"]
grouped.setdefault(keyword, []).append(item)
return grouped
def generate_script_for_topic(
client: OpenAI,
keyword: str,
videos: list[dict],
) -> str:
"""Generate a narration script for a topic using the LLM."""
# Take the top 15 descriptions to stay within token limits
descriptions = [v["description"] for v in videos[:15] if v["description"]]
descriptions_text = "\n".join(
f"- {desc}" for desc in descriptions
)
user_prompt = (
f"Topic: {keyword.replace('_', ' ')}\n\n"
f"Video descriptions from the field:\n{descriptions_text}\n\n"
f"Write the news narration script."
)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
],
temperature=0.7,
max_tokens=500,
)
return response.choices[0].message.content.strip()
def main():
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
grouped = load_metadata()
SCRIPTS_DIR.mkdir(parents=True, exist_ok=True)
for keyword, videos in grouped.items():
print(f"✍️ Generating script for: {keyword}...")
script = generate_script_for_topic(client, keyword, videos)
script_path = SCRIPTS_DIR / f"{keyword}_script.txt"
script_path.write_text(script, encoding="utf-8")
print(f" 📝 Saved to {script_path}")
print(f" Preview: {script[:120]}...\n")
if __name__ == "__main__":
main()
Create generate_tts.py:
# generate_tts.py
import os
from pathlib import Path
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
SCRIPTS_DIR = Path("data/scripts")
AUDIO_DIR = Path("data/audio")
def main():
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
AUDIO_DIR.mkdir(parents=True, exist_ok=True)
script_files = sorted(SCRIPTS_DIR.glob("*_script.txt"))
for script_path in script_files:
keyword = script_path.stem.replace("_script", "")
audio_path = AUDIO_DIR / f"{keyword}_narration.mp3"
if audio_path.exists():
print(f"⏭️ Skipping {keyword} (audio exists)")
continue
print(f"🔊 Generating TTS for: {keyword}...")
script_text = script_path.read_text(encoding="utf-8")
response = client.audio.speech.create(
model="tts-1",
voice="onyx", # Deep, authoritative news voice
input=script_text,
)
response.stream_to_file(str(audio_path))
print(f" ✅ Saved to {audio_path}")
if __name__ == "__main__":
main()
Now we bring it all together — combine downloaded video clips with the AI narration audio into a single news segment.
Create compile_video.py:
# compile_video.py
import json
import subprocess
from pathlib import Path
VIDEOS_DIR = Path("data/videos")
AUDIO_DIR = Path("data/audio")
OUTPUT_DIR = Path("data/output")
METADATA_PATH = VIDEOS_DIR / "metadata_index.json"
def get_video_duration(video_path: str) -> float:
"""Get video duration in seconds using ffprobe."""
result = subprocess.run(
[
"ffprobe", "-v", "error",
"-show_entries", "format=duration",
"-of", "default=noprint_wrappers=1:nokey=1",
video_path,
],
capture_output=True, text=True,
)
return float(result.stdout.strip())
def compile_topic(keyword: str, video_paths: list[str]):
"""Compile videos for a single topic into one news segment."""
audio_path = AUDIO_DIR / f"{keyword}_narration.mp3"
if not audio_path.exists():
print(f"⚠️ No narration audio for {keyword}, skipping.")
return
# Filter to videos that actually exist and take the top 5
existing = [p for p in video_paths if Path(p).exists()][:5]
if not existing:
print(f"⚠️ No videos found for {keyword}, skipping.")
return
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
# Create a concat file for FFmpeg
concat_file = OUTPUT_DIR / f"{keyword}_concat.txt"
with open(concat_file, "w") as f:
for vp in existing:
f.write(f"file '{Path(vp).absolute()}'\n")
output_path = OUTPUT_DIR / f"{keyword}_news_segment.mp4"
# Step 1: Concatenate video clips
temp_video = OUTPUT_DIR / f"{keyword}_temp_concat.mp4"
subprocess.run([
"ffmpeg", "-y",
"-f", "concat", "-safe", "0",
"-i", str(concat_file),
"-c:v", "libx264",
"-crf", "23",
"-preset", "fast",
"-vf", "scale=1920:1080:force_original_aspect_ratio=decrease,"
"pad=1920:1080:(ow-iw)/2:(oh-ih)/2",
"-r", "30",
"-an", # Remove original audio
str(temp_video),
], check=True)
# Step 2: Overlay narration audio
subprocess.run([
"ffmpeg", "-y",
"-i", str(temp_video),
"-i", str(audio_path),
"-c:v", "copy",
"-c:a", "aac",
"-b:a", "192k",
"-shortest",
str(output_path),
], check=True)
# Cleanup
temp_video.unlink(missing_ok=True)
concat_file.unlink(missing_ok=True)
duration = get_video_duration(str(output_path))
print(f"🎬 {keyword}: {output_path} ({duration:.1f}s)")
def main():
with open(METADATA_PATH, "r", encoding="utf-8") as f:
metadata = json.load(f)
# Group by keyword
grouped = {}
for item in metadata:
keyword = item["keyword"]
grouped.setdefault(keyword, []).append(item["local_path"])
for keyword, paths in grouped.items():
print(f"\n🎞️ Compiling: {keyword}")
compile_topic(keyword, paths)
print(f"\n✅ All segments compiled in {OUTPUT_DIR}/")
if __name__ == "__main__":
main()
# Generate narration scripts
python generate_script.py
# Convert to speech
python generate_tts.py
# Compile final videos
python compile_video.py
Expected output:
✍️ Generating script for: Ukraine_war...
📝 Saved to data/scripts/Ukraine_war_script.txt
🔊 Generating TTS for: Ukraine_war...
✅ Saved to data/audio/Ukraine_war_narration.mp3
🎞️ Compiling: Ukraine_war
🎬 Ukraine_war: data/output/Ukraine_war_news_segment.mp4 (87.3s)
✅ All segments compiled in data/output/
In Part 4, we’ll upload these compiled news segments to YouTube using the YouTube Data API, and set up Apify scheduling to run this entire pipeline automatically every 12 hours.
Looking for more TikTok data sources? Beyond keyword search, you can also pull trending videos, user profiles, hashtags, and comments. Browse our full collection of TikTok and Twitter scraping tools to expand your data pipeline.
]]>Series Navigation
- Part 1: Searching TikTok with the Apify Python Client
- Part 2: Downloading & Organizing Video Files
- Part 3: Auto-Generating News Scripts with AI ← You are here
- Part 4: Scheduling & Publishing to YouTube
Series Navigation
- Part 1: Searching TikTok with the Apify Python Client
- Part 2: Downloading & Organizing Video Files ← You are here
- Part 3: Auto-Generating News Scripts with AI
- Part 4: Scheduling & Publishing to YouTube
pip install httpx tqdm
We’ll use httpx for its first-class async support and tqdm for progress bars.
Create download_videos.py:
# download_videos.py
import asyncio
import json
import re
import sys
from pathlib import Path
import httpx
from tqdm.asyncio import tqdm_asyncio
MAX_CONCURRENT = 5
DOWNLOAD_DIR = Path("data/videos")
TIMEOUT = 60.0
def sanitize_filename(name: str) -> str:
"""Remove unsafe characters from a filename."""
return re.sub(r'[^\w\-.]', '_', name)[:80]
def load_results(json_path: str) -> list[dict]:
"""Load the search results JSON file."""
with open(json_path, "r", encoding="utf-8") as f:
return json.load(f)
def build_download_manifest(results: list[dict]) -> list[dict]:
"""Build a list of downloads, skipping entries without a valid URL."""
manifest = []
seen_ids = set()
for item in results:
aweme_id = item.get("aweme_id")
url = item.get("video_url_no_watermark")
if not aweme_id or not url:
continue
if aweme_id in seen_ids:
continue
seen_ids.add(aweme_id)
keyword = sanitize_filename(item.get("_search_keyword", "unknown"))
filename = f"{aweme_id}.mp4"
manifest.append({
"aweme_id": aweme_id,
"url": url,
"keyword": keyword,
"output_dir": DOWNLOAD_DIR / keyword,
"output_path": DOWNLOAD_DIR / keyword / filename,
"description": item.get("description", ""),
})
return manifest
async def download_one(
client: httpx.AsyncClient,
task: dict,
semaphore: asyncio.Semaphore,
) -> dict:
"""Download a single video file with retry logic."""
task["output_dir"].mkdir(parents=True, exist_ok=True)
if task["output_path"].exists():
return {"aweme_id": task["aweme_id"], "status": "skipped"}
async with semaphore:
for attempt in range(3):
try:
async with client.stream("GET", task["url"],
timeout=TIMEOUT) as resp:
resp.raise_for_status()
with open(task["output_path"], "wb") as f:
async for chunk in resp.aiter_bytes(
chunk_size=1024 * 64
):
f.write(chunk)
return {"aweme_id": task["aweme_id"], "status": "ok"}
except (httpx.HTTPStatusError, httpx.ReadTimeout) as e:
if attempt == 2:
return {
"aweme_id": task["aweme_id"],
"status": "failed",
"error": str(e),
}
await asyncio.sleep(2 ** attempt)
return {"aweme_id": task["aweme_id"], "status": "failed"}
async def download_all(manifest: list[dict]) -> list[dict]:
"""Download all videos concurrently with a semaphore limit."""
semaphore = asyncio.Semaphore(MAX_CONCURRENT)
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36"
),
"Referer": "https://www.tiktok.com/",
}
async with httpx.AsyncClient(
headers=headers, follow_redirects=True
) as client:
tasks = [
download_one(client, task, semaphore)
for task in manifest
]
results = await tqdm_asyncio.gather(
*tasks, desc="Downloading videos"
)
return results
def save_metadata(manifest: list[dict], output_path: Path):
"""Save a metadata index alongside the downloaded videos."""
metadata = []
for task in manifest:
metadata.append({
"aweme_id": task["aweme_id"],
"keyword": task["keyword"],
"description": task["description"],
"local_path": str(task["output_path"]),
})
with open(output_path, "w", encoding="utf-8") as f:
json.dump(metadata, f, ensure_ascii=False, indent=2)
def main():
if len(sys.argv) < 2:
print("Usage: python download_videos.py <path_to_results.json>")
sys.exit(1)
json_path = sys.argv[1]
results = load_results(json_path)
manifest = build_download_manifest(results)
print(f"📋 {len(manifest)} unique videos to download "
f"(from {len(results)} search results)")
download_results = asyncio.run(download_all(manifest))
ok = sum(1 for r in download_results if r["status"] == "ok")
skipped = sum(1 for r in download_results if r["status"] == "skipped")
failed = sum(1 for r in download_results if r["status"] == "failed")
print(f"\n✅ Downloaded: {ok} ⏭️ Skipped: {skipped} ❌ Failed: {failed}")
# Save metadata index
meta_path = DOWNLOAD_DIR / "metadata_index.json"
save_metadata(manifest, meta_path)
print(f"📁 Metadata saved to {meta_path}")
if __name__ == "__main__":
main()
python download_videos.py data/raw/tiktok_results_20260324_080000.json
Expected output:
📋 118 unique videos to download (from 122 search results)
Downloading videos: 100%|████████████████| 118/118 [01:42<00:00]
✅ Downloaded: 112 ⏭️ Skipped: 0 ❌ Failed: 6
📁 Metadata saved to data/videos/metadata_index.json
data/
└── videos/
├── metadata_index.json
├── Ukraine_war/
│ ├── 7229167805625847041.mp4
│ ├── 7229167805625847042.mp4
│ └── ...
├── earthquake_Turkey/
│ ├── 7229167805625847050.mp4
│ └── ...
└── protest_France/
├── 7229167805625847060.mp4
└── ...
Videos are automatically grouped by keyword, making it easy to find footage for a specific news topic.
TikTok video CDN URLs are short-lived and can be slow from certain regions. Downloading 100+ videos sequentially could take 30+ minutes. With asyncio and a concurrency limit of 5, we typically finish in under 3 minutes.
The build_download_manifest function tracks aweme_id values in a set, ensuring we never download the same video twice — even if it appears in multiple search results.
CDN URLs can occasionally return 403 or timeout. Our downloader retries up to 3 times with exponential backoff (1s, 2s, 4s) before marking a video as failed.
The metadata_index.json file serves as a manifest for the next steps in the pipeline. It maps each video to its local path, original keyword, and description — exactly the data we’ll need to generate narration scripts in Part 3.
In Part 3, we’ll use the downloaded videos and their descriptions to auto-generate news narration scripts with AI (OpenAI / Gemini), convert them to speech, and compile everything into a single news segment using FFmpeg.
]]>Series Navigation
- Part 1: Searching TikTok with the Apify Python Client
- Part 2: Downloading & Organizing Video Files ← You are here
- Part 3: Auto-Generating News Scripts with AI
- Part 4: Scheduling & Publishing to YouTube
Series Navigation
- Part 1: Searching TikTok with the Apify Python Client ← You are here
- Part 2: Downloading & Organizing Video Files
- Part 3: Auto-Generating News Scripts with AI
- Part 4: Scheduling & Publishing to YouTube
pip install apify-client python-dotenv
Create a .env file in your project root:
APIFY_TOKEN=your_apify_api_token_here
Create a file called config.py to centralize your search parameters. This makes it easy to manage multiple hotspot topics:
# config.py
HOTSPOT_SEARCHES = [
{
"keyword": "Ukraine war",
"region": "UA",
"sortType": 2, # Most recent
"publishTime": "WEEK",
"limit": 50,
},
{
"keyword": "earthquake Turkey",
"region": "TR",
"sortType": 2,
"publishTime": "YESTERDAY",
"limit": 50,
},
{
"keyword": "protest France",
"region": "FR",
"sortType": 1, # Most liked
"publishTime": "WEEK",
"limit": 30,
},
]
Each dictionary maps directly to the input schema of the Advanced TikTok Search API Actor. The key parameters are:
| Parameter | Purpose |
|---|---|
keyword |
The search term (supports spaces) |
region |
Two-letter country code to localize results |
sortType |
0 = Relevance, 1 = Most Liked, 2 = Most Recent |
publishTime |
Time filter: YESTERDAY, WEEK, MONTH, etc. |
limit |
Soft cap on the number of videos returned |
Create search_tiktok.py:
# search_tiktok.py
import json
import os
from datetime import datetime
from pathlib import Path
from apify_client import ApifyClient
from dotenv import load_dotenv
from config import HOTSPOT_SEARCHES
load_dotenv()
ACTOR_ID = "novi/advanced-search-tiktok-api"
OUTPUT_DIR = Path("data/raw")
def run_search(client: ApifyClient, search_params: dict) -> list[dict]:
"""Run a single TikTok search and return the results."""
print(f"🔍 Searching: '{search_params['keyword']}' "
f"(region={search_params.get('region', 'ANY')})...")
run = client.actor(ACTOR_ID).call(run_input=search_params)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f" ✅ Found {len(items)} videos.")
return items
def extract_essential_fields(video: dict) -> dict:
"""Extract only the fields we need for the pipeline."""
stats = video.get("statistics", {})
author = video.get("author", {})
play_addr = video.get("video", {}).get("play_addr", {})
download_addr = video.get("video", {}).get("download_addr", {})
return {
"aweme_id": video.get("aweme_id"),
"description": video.get("desc", ""),
"share_url": video.get("share_url", ""),
"create_time": video.get("create_time"),
"region": video.get("region"),
# Author info
"author_username": author.get("unique_id"),
"author_nickname": author.get("nickname"),
# Statistics
"play_count": stats.get("play_count", 0),
"like_count": stats.get("digg_count", 0),
"comment_count": stats.get("comment_count", 0),
"share_count": stats.get("share_count", 0),
# Video URLs
"video_url_no_watermark": (
play_addr.get("url_list", [None])[0]
),
"video_url_watermark": (
download_addr.get("url_list", [None])[0]
),
"video_duration_ms": video.get("video", {}).get("duration"),
# Hashtags
"hashtags": [
t.get("hashtag_name", "")
for t in video.get("text_extra", [])
if t.get("type") == 1
],
}
def main():
token = os.environ.get("APIFY_TOKEN")
if not token:
raise SystemExit("❌ APIFY_TOKEN not set. Check your .env file.")
client = ApifyClient(token)
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
all_results = []
for search_params in HOTSPOT_SEARCHES:
raw_videos = run_search(client, search_params)
extracted = [extract_essential_fields(v) for v in raw_videos]
# Tag each result with the original search keyword
for item in extracted:
item["_search_keyword"] = search_params["keyword"]
all_results.extend(extracted)
# Save to a single JSON file
output_path = OUTPUT_DIR / f"tiktok_results_{timestamp}.json"
with open(output_path, "w", encoding="utf-8") as f:
json.dump(all_results, f, ensure_ascii=False, indent=2)
print(f"\n📁 Saved {len(all_results)} videos to {output_path}")
if __name__ == "__main__":
main()
python search_tiktok.py
Expected output:
🔍 Searching: 'Ukraine war' (region=UA)...
✅ Found 50 videos.
🔍 Searching: 'earthquake Turkey' (region=TR)...
✅ Found 42 videos.
🔍 Searching: 'protest France' (region=FR)...
✅ Found 30 videos.
📁 Saved 122 videos to data/raw/tiktok_results_20260324_080000.json
Each entry in the JSON file looks like this:
{
"aweme_id": "7229167805625847041",
"description": "Breaking: massive protest in central Paris #protest #france",
"share_url": "https://www.tiktok.com/@user/video/7229167805625847041",
"create_time": 1683171893,
"region": "FR",
"author_username": "news_reporter_01",
"author_nickname": "Breaking News FR",
"play_count": 585709,
"like_count": 25006,
"comment_count": 183,
"share_count": 492,
"video_url_no_watermark": "https://v19.tiktokcdn-us.com/...",
"video_url_watermark": "https://v19.tiktokcdn-us.com/...",
"video_duration_ms": 54635,
"hashtags": ["protest", "france"],
"_search_keyword": "protest France"
}
The video_url_no_watermark field is the key asset — this is the clean video URL we’ll download in Part 2.
In Part 2, we’ll write an async downloader that fetches all these watermark-free videos, organizes them by topic and date, and handles retries for failed downloads.
]]>Series Navigation
- Part 1: Searching TikTok with the Apify Python Client ← You are here
- Part 2: Downloading & Organizing Video Files
- Part 3: Auto-Generating News Scripts with AI
- Part 4: Scheduling & Publishing to YouTube
This is where the Advanced TikTok Search API Actor on Apify comes into play. In this post, we’ll explore an innovative idea: automating the curation of a YouTube news channel using this powerful web scraping tool.
The concept is straightforward: creating a YouTube channel (using YouTube Shorts and long-form compilations) that reports on current global hot spots. The content relies on citizen journalism and on-the-ground footage sourced directly from TikTok.
To execute this, you need a reliable pipeline to:
The Advanced TikTok Search API Actor is uniquely positioned to handle the data collection phase of this pipeline. Let’s look at how its features align with our content creation strategy.
When covering global hot spots, recency is everything. The actor allows you to use the publishTime filter. By setting it to YESTERDAY or WEEK, you ensure that you are only scraping the most recent footage of an ongoing event.
Furthermore, the region parameter lets you target specific countries using their two-letter code (e.g., UA for Ukraine, SD for Sudan). This is crucial for verifying that the footage is likely originating from the hotspot itself, filtering out irrelevant global noise.
You can automate searches using specific, localized keywords. The actor’s keyword parameter is perfect for this. Once you have a search phrase, the sortType parameter becomes incredibly useful:
sortType: 2 (Most recent) gives you a chronological feed of breaking events.sortType: 1 (Most liked) helps you find the most viral, impactful footage that is already resonating with audiences.A professional YouTube news channel cannot have distracting TikTok watermarks bouncing around the screen. The Advanced TikTok Search API Actor provides access to both play_addr (usually watermark-free) and download_addr (with watermark) in its JSON output. Accessing the clean video URL allows for seamless integration into video editing software like Premiere Pro or automated editing pipelines.
For major events, you might need hundreds of clips to find the perfect angles. Using the limit and isUnlimited: true parameters, you can instruct the actor to dig deep into the search results. While unlimited scraping takes more time and resources, it provides a comprehensive dataset of videos for your editorial team (or automated scripts) to review.
Here’s a high-level view of how you could automate this YouTube channel using Apify and other tools:
.mp4 files from the provided URLs. You can then use tools like FFmpeg to concatenate clips, add a standard channel intro/outro, or even integrate AI text-to-speech APIs to read auto-generated news scripts based on the video captions (desc field).Building a news channel around global hot spots requires speed, accuracy, and access to raw footage. By using the Advanced TikTok Search API Actor to systematically scrape and filter TikTok data, you can build a highly efficient news aggregation pipeline. It transforms the chaotic landscape of social media into a structured, easily accessible database of breaking news, ready to be edited and delivered to a global YouTube audience.
Ready to build this pipeline? Follow our step-by-step Python implementation series:
]]>