Open Meshy Projekt

OpenMeshy ist ein experimentelles Open-Source-Projekt zur lokalen Generierung von 3D-Modellen. Inspiriert von kommerziellen Lösungen wie Meshy.ai, liegt der Fokus hier auf vollständiger Unabhängigkeit und Kostenfreiheit durch die Nutzung moderner KI-Modelle auf eigener Hardware.

Das Projekt kombiniert führende Open-Source-Frameworks zu einer leistungsstarken Pipeline:

XAMPP: Bereitstellung der Weboberfläche
Python: Die zentrale Steuerung und Logik der Anwendung.
Stable Diffusion: Erstellung präziser 2D-Referenzbilder und Texturen aus Prompts.
TripoSR: Blitzschnelle Rekonstruktion von 3D-Strukturen aus den generierten Bildern.

Important: This documentation is currently a work in progress and may be incomplete.
Suggestions for improvement are welcome and can be submitted via the contact form.

Deutsch:

1. Grundvoraussetzungen installieren

Installiere das ALLES zuerst:

Python (3.10.x empfohlen)
- ⚠️ Wichtig: NICHT 3.11 oder 3.12
- Beim Installieren: Add to PATH
Git
- Wird für Repos gebraucht
CMake
- Wichtig für Open3D / native Builds
Microsoft Visual C++ Build Tools
- Workload: Desktop development with C++

2. XAMPP (Frontend Hosting)

Installiere XAMPP
Starte:
- Apache

Dein Webprojekt liegt dann z. B. hier:

C:\xampp\htdocs\openmeshy

3. Stable Diffusion (AUTOMATIC1111)

Installation

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
cd stable-diffusion-webui

Start-Konfiguration

Bearbeite webui-user.bat:

set COMMANDLINE_ARGS=--api --cors-allow-origins "*" --allow-code

Stable Diffusion muss noch konfiguriert werden, dazu müssen Modelle platziert werden.

Lade ein .safetensors– oder .ckpt-Modell herunter.

und kopiere es in den Ordner stable-diffusion-webui\models\Stable-diffusion\

Neben den Hauptmodellen (Checkpoints) gibt es auch LoRAs

Diese legst du in den entsprechenden Unterordnern im models-Verzeichnis ab

für genauere Anleitung schaue im Internet oder frage eine AI.

Stable Diffusion starten

webui-user.bat

API läuft auf:

http://127.0.0.1:7860

4. TripoSR (Image → 3D Mesh)

Installation

git clone https://github.com/VAST-AI-Research/TripoSR.git
cd TripoSR

Virtuelle Umgebung

python -m venv venv
venv\Scripts\activate

Dependencies

pip install -r requirements.txt
pip install rembg
pip install torch torchvision torchaudio
pip install open3d
pip install git+https://github.com/tatsy/torchmcubes.git
 

Falls GPU: passende CUDA-Version installieren

5. OpenMeshy Backend (FastAPI)

Das ist dein eigener Server.

Installation

pip install fastapi uvicorn
pip install pillow numpy opencv-python

pip install matplotlib

Installiere zusätzlich:

pip install rembg
pip install open3d
pip install torch
pip install requests

6. Projektstruktur

So könnte dein Setup ungefähr aussehen:

C:\OpenMeshy\
│
├── backend\
│   └── main.py (FastAPI)
│
├── \Users\DeinBenutzername\TripoSR\
│   ├── run.py
│   └── venv\
│
├── stable-diffusion-webui\
│
└── xampp\htdocs\openmeshy\
        └── index.html

7. Open Meshy vorbereiten

index.html und info.html in den htdocs/openmehsy Ordner kopieren

die Datei main.py in den Ordner C:\OpenMeshy kopieren

in der Datei main.py muss der Pfad zu deinem TripoSR Ordner geändert werden wenn er woanders als hier "C:\Users\DeinBenutzername\TripoSR" liegt

TRIPOSR_DIR = os.path.join(
os.path.expanduser("~"),
"TripoSR"
)

dieser Teil muss angepasst werden, je nachdem wo dein Ordner liegt

8. Backend starten

uvicorn main:app --reload

läuft dann auf:

http://127.0.0.1:8000

9. Aufrufen der Webseite im Browser

gebe folgendes in deinen Browser ein nachdem Xampp, main.py und Stable Diffusion gestartet wurde

http://127.0.0.1/openmeshy/

Important: This documentation is currently a work in progress and may be incomplete.
Suggestions for improvement are welcome and can be submitted via the contact form.

English:

1. Install Basic Requirements

Install ALL of this first:

Python (3.10.x recommended)
- ⚠️ Important: NOT 3.11 or 3.12
- During installation: Check Add to PATH
Git – needed for repositories
CMake – important for Open3D / native builds
Microsoft Visual C++ Build Tools
- Workload: Desktop development with C++

2. XAMPP (Frontend Hosting)

Install XAMPP
Start: Apache

Your web project will then be located here, for example:

C:\xampp\htdocs\openmeshy

3. Stable Diffusion (AUTOMATIC1111)

Installation

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
cd stable-diffusion-webui

Startup Configuration

Edit webui-user.bat:

set COMMANDLINE_ARGS=--api --cors-allow-origins "*" --allow-code

Stable Diffusion still needs to be configured – you must place models.

Download a .safetensors or .ckpt model
Copy it into the folder: stable-diffusion-webui\models\Stable-diffusion\

In addition to the main models (checkpoints), there are also LoRAs.

Place them in the corresponding subfolders inside the models directory.

For more detailed instructions, search online or ask an AI.

Start Stable Diffusion

webui-user.bat

The API runs on:

http://127.0.0.1:7860

4. TripoSR (Image → 3D Mesh)

Installation

git clone https://github.com/VAST-AI-Research/TripoSR.git
cd TripoSR

Virtual Environment

python -m venv venv
venv\Scripts\activate

Dependencies

pip install -r requirements.txt
pip install rembg
pip install torch torchvision torchaudio
pip install open3d
pip install git+https://github.com/tatsy/torchmcubes.git

If you have a GPU: install the appropriate CUDA version.

5. OpenMeshy Backend (FastAPI)

This is your own server.

Installation

pip install fastapi uvicorn
pip install pillow numpy opencv-python
pip install matplotlib

pip install rembg
pip install open3d
pip install torch
pip install requests

6. Project Structure

Your setup could look something like this:

C:\OpenMeshy\
│
├── backend\
│ └── main.py (FastAPI)
│
├── Users\YourUsername\TripoSR\
│ ├── run.py
│ └── venv\
│
├── stable-diffusion-webui\
│
└── xampp\htdocs\openmeshy\
└── index.html

7. Prepare Open Meshy

Copy index.html and info.html into the htdocs\openmeshy folder
Copy the main.py file into the C:\OpenMeshy folder

In the main.py file, the path to your TripoSR folder must be changed if it is located somewhere other than:

C:\Users\YourUsername\TripoSR

TRIPOSR_DIR = os.path.join(
os.path.expanduser("~"),
"TripoSR"
)

This part must be adjusted depending on where your folder is located.

8. Start the Backend

uvicorn main:app --reload

It then runs on:

http://127.0.0.1:8000

9. Open the Website in Your Browser

After XAMPP, main.py, and Stable Diffusion have been started, enter the following in your browser:

http://127.0.0.1/openmeshy/

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Open Meshy</title>

    <script type="importmap">
    {
      "imports": {
        "three": "https://cdn.jsdelivr.net/npm/three@0.158.0/build/three.module.js"
      }
    }
    </script>

    <style>
        body {
            font-family: Arial, sans-serif;
            background: #0f0f0f;
            color: #ffffff;
            display: flex;
            justify-content: center;
            margin: 0;
            padding-top: 80px;
        }

        .container {
            width: 420px;
            padding: 25px;
            background: #1a1a1a;
            border-radius: 12px;
            box-shadow: 0 0 25px rgba(0,0,0,0.6);
        }

        .texture-btn {
            background: #2196F3 !important; /* Blue for texture-related actions */
            margin-top: 10px !important;
        }

        .texture-btn:hover {
            background: #1976D2 !important;
        }

        .texture-preview {
            width: 100%;
            max-height: 250px;
            object-fit: cover;
            border-radius: 8px;
            margin-top: 5px;
            border: 1px dashed #444;
            display: none; /* Initially hidden until processing is complete */
        }

        h1 {
            text-align: center;
            margin-bottom: 10px;
        }

        p {
            text-align: center;
            font-size: 14px;
            opacity: 0.7;
        }

        label {
            font-size: 14px;
        }

        textarea {
            width: 100%;
            height: 80px;
            margin-bottom: 10px;
            padding: 10px;
            border-radius: 6px;
            border: none;
            resize: none;
            background: #2a2a2a;
            color: white;
            box-sizing: border-box; /* Ensures padding doesn't affect width */
        }

        button {
            width: 100%;
            padding: 12px;
            background: #00c853;
            border: none;
            border-radius: 6px;
            color: white;
            font-size: 16px;
            cursor: pointer;
            margin-top: 10px;
            transition: 0.2s;
        }

        button:hover {
            background: #00e676;
        }

        button:disabled {
            background: #555;
            cursor: not-allowed;
        }

        .status {
            margin-top: 10px;
            text-align: center;
            font-size: 14px;
            opacity: 0.8;
        }

        #resultImage {
            margin-top: 15px;
            width: 100%;
            max-height: 300px;
            object-fit: contain;
            border-radius: 8px;
            display: none;
        }

        /* Loading Overlay Styling */
        #overlay {
            display: none;
            position: fixed;
            top: 0; left: 0;
            width: 100%; height: 100%;
            background: rgba(0,0,0,0.75);
            color: white;
            justify-content: center;
            align-items: center;
            font-size: 20px;
            z-index: 999;
        }

        .navbar {
            position: fixed;
            top: 0;
            left: 0;
            width: 100%;
            height: 60px;
            background: #141414;
            display: flex;
            justify-content: space-between;
            align-items: center;
            padding: 0 25px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.6);
            z-index: 1000;
        }

        .logo {
            font-weight: bold;
            font-size: 18px;
        }

        .nav-right {
            position: absolute;
            left: 50%;
            transform: translateX(-50%);
            display: flex;
            gap: 20px;
        }

        .nav-right a {
            color: #ffffff;
            text-decoration: none;
            margin-left: 20px;
            font-size: 14px;
            opacity: 0.8;
            transition: 0.2s;
        }

        .nav-right a:hover {
            opacity: 1;
            color: #00e676;
        }

        .download-btn {
            background: #2a2a2a !important;
            margin-top: 5px !important;
            font-size: 14px !important;
            padding: 8px !important;
        }

        .download-btn:hover {
            background: #3a3a3a !important;
        }
    </style>
</head>
<body>

<div class="navbar">
    <div class="nav-left">
        <span class="logo">🧠 Open Meshy</span>
    </div>

    <div class="nav-right">
        <a href="info.html">About</a>
        <a href="https://computerkids.berlin/kontakt/">Contact</a>
    </div>
</div>

<div id="overlay">
    Generating... please wait ⏳
</div>

<div class="container">
    <h1>🧠 Open Meshy</h1>
    <p>Step 1: Generate Image → Step 2: Create 3D Mesh</p>

    <label>Prompt</label>
    <textarea id="prompt" placeholder="e.g., low poly tank, green, stylized"></textarea>

    <label>Negative Prompt</label>
    <textarea id="negative" placeholder="e.g., blurry, realistic, high detail"></textarea>

    <button onclick="generate()">Generate Image</button>

    <div class="status" id="status">System Ready</div>

    <input type="file" id="fileInput" accept="image/*" style="display: none;" />
    <button onclick="uploadImage()">Upload Existing Image</button>

    <img id="resultImage" />
    <img id="depthImage" style="display:none; width:100%; margin-top:10px; border-radius:8px;" />

    <div id="viewer" style="width:100%; height:300px; margin-top:15px; display:none;"></div>

    <button id="downloadBtn" style="display:none;">Download Mesh</button>

    <button id="continueBtn" disabled onclick="continueProcess()">
        Continue / Create 3D Mesh
    </button>

    <div id="resultsArea" style="display:none; margin-top: 20px;">
        
        <h3 style="text-align:center; color:#00c853;">Generation Results</h3>

        <div style="display: flex; gap: 10px; margin-bottom: 20px;">
            <div style="flex: 1; text-align: center;">
                <label style="font-size:12px; opacity:0.7;">Original View</label>
                <img id="resultImage" style="width:100%; max-height:200px; object-fit:contain; border-radius:8px;" />
            </div>
            <div style="flex: 1; text-align: center;">
                <label style="font-size:12px; opacity:0.7;">Depth Map</label>
                <img id="depthImage" style="width:100%; max-height:200px; object-fit:contain; border-radius:8px;" />
            </div>
        </div>

        <div style="margin-bottom: 20px;">
            <label>3D Mesh (from Original Image)</label>
            <div id="viewerOriginal" style="width:100%; height:250px; background:#141414; border-radius:8px; margin-top:5px;"></div>
            <button id="downloadOriginalBtn" class="download-btn">⬇️ Download Original Mesh</button>
        </div>

        <div style="margin-bottom: 20px;">
            <label>3D Mesh (from Depth Map)</label>
            <div id="viewerDepth" style="width:100%; height:250px; background:#141414; border-radius:8px; margin-top:5px;"></div>
            <button id="downloadDepthBtn" class="download-btn">⬇️ Download Depth Mesh</button>
        </div>

        <div style="margin-bottom: 20px; border-top: 1px solid #333; padding-top: 20px;">
            <label>🎨 Texture Generation (img2img)</label>
            <p style="text-align:left; font-size:12px; margin-bottom:10px;">
                Creates a matching, flat texture based on the original image using AI.
            </p>
            <img id="texturePreview" class="texture-preview" />
            <button id="genTextureBtn" class="texture-btn" onclick="generateTexture()">
                ✨ Generate Textures
            </button>
            <button id="downloadTextureBtn" class="download-btn" style="display:none;">
                ⬇️ Download Texture (.png)
            </button>
        </div>
    </div>
</div>

<script type="module">
    import * as THREE from "https://cdn.jsdelivr.net/npm/three@0.158.0/build/three.module.js";
    import { OBJLoader } from "https://cdn.jsdelivr.net/npm/three@0.158.0/examples/jsm/loaders/OBJLoader.js";
    import { OrbitControls } from "https://cdn.jsdelivr.net/npm/three@0.158.0/examples/jsm/controls/OrbitControls.js";

/**
 * Stage 1: Generates an AI image based on the text prompt.
 * Communicates with the local server to trigger the diffusion model.
 */
async function generate() {
    const prompt = document.getElementById("prompt").value;
    const negative = document.getElementById("negative").value;
    const status = document.getElementById("status");
    const overlay = document.getElementById("overlay");

    if (!prompt) {
        status.innerText = "Please enter a prompt!";
        alert("Prompt is required to generate an image.");
        return;
    }

    status.innerText = "Generating image...";
    overlay.style.display = "flex";

    try {
        const response = await fetch("http://localhost:8000/generate", {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({
                prompt: prompt,
                negative_prompt: negative
            })
        });

        if (!response.ok) throw new Error("Server error occurred during generation.");

        const data = await response.json();

        if (data.images && data.images.length > 0) {
            const base64 = data.images[0];
            const imageUrl = "data:image/png;base64," + base64;

            const img = document.getElementById("resultImage");
            img.src = imageUrl;
            img.style.display = "block";

            status.innerText = "Image generated successfully!";
            document.getElementById("continueBtn").disabled = false;
        } else {
            status.innerText = "No image data received from server.";
        }
    } catch (error) {
        console.error(error);
        status.innerText = "Error: Is your local backend running?";
    }

    overlay.style.display = "none";
}

/**
 * Stage 2: Takes the generated image and requests a 3D Mesh conversion.
 * Sends the Base64 image back to the server for OBJ creation.
 */
function continueProcess() {
    const img = document.getElementById("resultImage");
    const src = img.src; 
    
    const status = document.getElementById("status");
    const overlay = document.getElementById("overlay");

    if (!src || src.startsWith('data:image/png;base64,,')) {
        alert("No valid image found! Please generate an image first.");
        return;
    }

    status.innerText = "Creating 3D Meshes (this may take a moment)...";
    overlay.style.display = "flex";
    
    // Clear previous viewer instances
    document.getElementById("viewerOriginal").innerHTML = "";
    document.getElementById("viewerDepth").innerHTML = "";

    fetch("http://localhost:8000/mesh", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ image: src })
    })
    .then(res => res.json())
    .then(data => {
        console.log("Mesh Response:", data);
        if (data.status === "ok" || data.run_id) {
            pollStatus(data.run_id); // Start checking the processing status
        } else {
            overlay.style.display = "none";
            status.innerText = "Mesh generation failed.";
            alert("Backend Error:\n" + JSON.stringify(data, null, 2));
        }
    })
    .catch(err => {
        console.error("NETWORK OR JS ERROR:", err);
        overlay.style.display = "none";
        status.innerText = "Frontend error (check console).";
    });
}

/**
 * Initializes a Three.js scene to preview the generated OBJ models.
 * @param {string} objPath - The URL to the .obj file.
 * @param {string} containerId - The ID of the HTML element to render into.
 */
function show3DModel(objPath, containerId) {
    const status = document.getElementById("status");
    const viewer = document.getElementById(containerId);
    if (!viewer) return;
    viewer.innerHTML = ""; 

    const scene = new THREE.Scene();
    const aspect = viewer.clientWidth / viewer.clientHeight;
    const camera = new THREE.PerspectiveCamera(75, aspect, 0.1, 1000);

    const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
    renderer.setSize(viewer.clientWidth, viewer.clientHeight);
    renderer.setClearColor(0x000000, 0); 
    viewer.appendChild(renderer.domElement);

    // Basic Lighting setup
    scene.add(new THREE.AmbientLight(0xffffff, 0.6));
    const light = new THREE.DirectionalLight(0xffffff, 1);
    light.position.set(1, 1, 1);
    scene.add(light);

    const loader = new OBJLoader();
    status.innerText = `Loading 3D model into ${containerId}...`;

    loader.load(objPath, function (object) {
        // Apply a neutral standard material to the mesh
        object.traverse(function (child) {
            if (child.isMesh) {
                child.material = new THREE.MeshStandardMaterial({ 
                    color: 0xffffff, 
                    metalness: 0.1,
                    roughness: 0.5
                });
            }
        });

        // Center the model in the scene
        const box = new THREE.Box3().setFromObject(object);
        const center = box.getCenter(new THREE.Vector3());
        object.position.sub(center);

        // Normalize scale for the preview window
        const size = box.getSize(new THREE.Vector3()).length();
        const scale = 2.5 / size; 
        object.scale.setScalar(scale);

        scene.add(object);
        status.innerText = "Model loaded successfully.";
    }, 
    // Progress logging
    function (xhr) {
        console.log(containerId + ': ' + (xhr.loaded / xhr.total * 100) + '% loaded');
    },
    // Error handling for loader
    function (error) {
        console.error('Error loading model:', error);
        viewer.innerHTML = "<p style='color:red; text-align:center; padding-top:100px;'>Failed to load 3D model.</p>";
    });

    const controls = new OrbitControls(camera, renderer.domElement);
    camera.position.set(0, 1, 4); 
    controls.update();

    function animate() {
        requestAnimationFrame(animate);
        controls.update();
        renderer.render(scene, camera);
    }
    animate();

    // Handle window resizing to keep the aspect ratio correct
    window.addEventListener('resize', () => {
        camera.aspect = viewer.clientWidth / viewer.clientHeight;
        camera.updateProjectionMatrix();
        renderer.setSize(viewer.clientWidth, viewer.clientHeight);
    }, false);
}

/**
 * Stage 3: Generates a texture map from the original image using img2img.
 */
async function generateTexture() {
    const status = document.getElementById("status");
    const overlay = document.getElementById("overlay");
    const resultImg = document.getElementById("resultImage");
    const texturePreview = document.getElementById("texturePreview");
    const downloadBtn = document.getElementById("downloadTextureBtn");

    if (!resultImg.src || resultImg.src.includes('placeholder')) {
        alert("No image found to create a texture from!");
        return;
    }

    status.innerText = "Extracting texture via img2img...";
    overlay.style.display = "flex";

    try {
        // Convert the current image source to a blob to avoid cross-origin or data-url issues
        const responseBlob = await fetch(resultImg.src);
        const blob = await responseBlob.blob();

        const base64data = await new Promise((resolve, reject) => {
            const reader = new FileReader();
            reader.onloadend = () => resolve(reader.result);
            reader.onerror = reject;
            reader.readAsDataURL(blob);
        });

        status.innerText = "AI is generating the texture...";

        const response = await fetch("http://localhost:8000/generate-texture", {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({
                image: base64data,
                denoising: 0.5
            })
        });

        const data = await response.json();

        if (data.status === "ok") {
            const imageUrl = "http://localhost:8000" + data.texture_url;
            texturePreview.src = imageUrl
            texturePreview.style.display = "block";
            
            downloadBtn.style.display = "block";
            downloadBtn.onclick = async () => {
                try {
                    // Download mechanism using a virtual link and Blob URL
                    const response = await fetch(texturePreview.src);
                    const blob = await response.blob();
                    const url = window.URL.createObjectURL(blob);
                    const link = document.createElement('a');
                    link.href = url;
                    link.download = "open_meshy_texture.png";
                    document.body.appendChild(link);
                    link.click();
                    document.body.removeChild(link);
                    window.URL.revokeObjectURL(url);
                } catch (err) {
                    console.error("Download failed:", err);
                    window.open(texturePreview.src, "_blank");
                }
            };
            
            status.innerText = "Texture ready!";
        } else {
            throw new Error(data.message || "Backend error during texture generation.");
        }
    } catch (error) {
        console.error(error);
        alert("Texture Error: " + error.message);
        status.innerText = "Failed to generate texture.";
    } finally {
        overlay.style.display = "none";
    }
}

/**
 * Utility: Triggers the hidden file input to allow users to upload their own images.
 */
function uploadImage() {
    const input = document.getElementById("fileInput");
    input.click();

    input.onchange = async () => {
        const file = input.files[0];
        if (!file) return;

        const formData = new FormData();
        formData.append("image", file);

        try {
            const response = await fetch("http://127.0.0.1:8000/upload", {
                method: "POST",
                body: formData
            });

            const result = await response.json();
            if (result.run_id) {
                document.getElementById("overlay").style.display = "flex";
                document.getElementById("status").innerText = "Upload successful, polling status...";
                pollStatus(result.run_id);
            }
        } catch (error) {
            console.error("Upload failed:", error);
            alert("Error uploading the image.");
        }
    };
}

/**
 * Polling Mechanism: Periodically checks the server for the completion status of a background task.
 * @param {string} runId - The ID of the current background process.
 */
async function pollStatus(runId) {
    const statusText = document.getElementById("status");

    while (true) {
        try {
            const response = await fetch(`http://localhost:8000/status/${runId}`);
            const data = await response.json();

            if (data.status === "completed") {
                statusText.innerText = "Generation complete!";
                displayResults(data); 
                document.getElementById("overlay").style.display = "none";
                break; 
            } else if (data.status === "error") {
                statusText.innerText = "Backend error: " + data.error;
                document.getElementById("overlay").style.display = "none";
                break;
            } else {
                statusText.innerText = "AI is working... (Creating 3D Models)";
            }
        } catch (err) {
            console.error("Polling error:", err);
        }

        // Wait 2 seconds before the next poll attempt
        await new Promise(resolve => setTimeout(resolve, 2000));
    }
}

/**
 * Updates the UI with final images and initializes the 3D viewers.
 * @param {object} data - The completion data containing URLs for images and meshes.
 */
function displayResults(data) {
    const resultsArea = document.getElementById("resultsArea");
    resultsArea.style.display = "block";

    const baseUrl = "http://localhost:8000";

    // Update result images
    document.getElementById("resultImage").src = baseUrl + data.original_image_url;
    document.getElementById("depthImage").src = baseUrl + data.depth_image_url;

    // Load 3D models into their respective viewers
    show3DModel(baseUrl + data.original_mesh_url, "viewerOriginal");
    show3DModel(baseUrl + data.depth_mesh_url, "viewerDepth");

    // Assign download links to buttons
    document.getElementById("downloadOriginalBtn").onclick = () => window.open(baseUrl + data.original_mesh_url, "_blank");
    document.getElementById("downloadDepthBtn").onclick = () => window.open(baseUrl + data.depth_mesh_url, "_blank");
}

// Global scope registration for HTML onclick attributes
window.uploadImage = uploadImage;
window.generate = generate;
window.continueProcess = continueProcess;
window.generateTexture = generateTexture;
</script>

</body>
</html>

index.html Herunterladen

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>OpenMeshy – Info</title>

    <style>
        body {
            font-family: Arial, sans-serif;
            background: #0f0f0f;
            color: #ffffff;
            margin: 0;
            padding-top: 80px;
            display: flex;
            justify-content: center;
        }

        .navbar {
            position: fixed;
            top: 0;
            left: 0;
            width: 100%;
            height: 60px;
            background: #141414;
            display: flex;
            align-items: center;
            justify-content: space-between;
            padding: 0 25px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.6);
            z-index: 1000;
        }

        .nav-links {
            position: absolute;
            left: 50%;
            transform: translateX(-50%);
            display: flex;
            gap: 20px;
        }

        .nav-links a {
            color: white;
            text-decoration: none;
            font-size: 14px;
            opacity: 0.8;
        }

        .nav-links a:hover {
            opacity: 1;
        }

        .container {
            width: 700px;
            background: #1a1a1a;
            padding: 30px;
            border-radius: 12px;
            box-shadow: 0 0 25px rgba(0,0,0,0.6);
        }

        h1 {
            text-align: center;
            margin-bottom: 20px;
        }

        h2 {
            color: #00e676;
            margin-top: 25px;
        }

        p {
            line-height: 1.6;
            opacity: 0.9;
        }

        ul {
            line-height: 1.6;
        }

    </style>
</head>
<body>

<div class="navbar">
    <div class="logo">🧠 OpenMeshy</div>

    <div class="nav-links">
        <a href="index.html">Home</a>
        <a href="https://computerkids.berlin/kontakt/">Contact</a>
    </div>
</div>

<div class="container">
    <h1>About OpenMeshy</h1>

    <h2>What is this?</h2>
    <p>
        OpenMeshy is an experimental open-source project for local 3D model generation.
        Inspired by commercial solutions like Meshy.ai, the focus here is on complete independence
        and cost-free usage by leveraging modern AI models on your own hardware.
    </p>

    <h2>Project Goals</h2>
    <ul>
        <li>Testing the limits of open-source technologies in 3D creation.</li>
        <li>Generating high-quality 3D assets without subscription models or cloud costs.</li>
        <li>Automating the pipeline from text input to the finished mesh.</li>
        <li>Creating a transparent alternative to proprietary AI platforms.</li>
    </ul>

    <h2>Technology & Requirements</h2>
    <p>
        The project combines leading open-source frameworks into a powerful pipeline:
    </p>
    <ul>
        <li><strong>XAMPP:</strong> Provides the web interface</li>
        <li><strong>Python:</strong> Central control and application logic.</li>
        <li><strong>Stable Diffusion:</strong> Creation of precise 2D reference images and textures from prompts.</li>
        <li><strong>TripoSR:</strong> Lightning-fast reconstruction of 3D structures from the generated images.</li>
    </ul>

    <h2>Why this project?</h2>
    <p>
        While commercial tools are often expensive and upload data to the cloud, OpenMeshy aims to show
        that professional 3D generation is possible locally and for free. It is a proof-of-concept
        for developers and creators who want to keep full control over their workflow.
    </p>
</div>
</body>
</html>

info.html Herunterladen

from fastapi import FastAPI, UploadFile, File, HTTPException, BackgroundTasks
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
import requests
import base64
import uuid
from fastapi import Request
from fastapi.responses import JSONResponse
import torch
import cv2
import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt
import threading
from fastapi.staticfiles import StaticFiles
import os
import subprocess
import sys
import shutil
import time
from PIL import Image
import io

# Define valid image formats for upload
ALLOWED_EXTENSIONS = {"png", "jpg", "jpeg"}

# Dictionary to store the status and results of asynchronous mesh generation tasks
mesh_tasks = {}

# Ensure required directories exist
os.makedirs("outputs", exist_ok=True)
if not os.path.exists("uploads"):
    os.makedirs("uploads")

# Path to the TripoSR local installation Directory
# IMPORTANT: Replace this with the path to your own TripoSR directory
TRIPOSR_DIR = os.path.join(
    os.path.expanduser("~"),
    "TripoSR"
)

# --- AI Model Initialization ---
# Load MiDaS model for depth estimation only once during startup to save memory/VRAM
print("Loading MiDaS depth estimation model...")
model = torch.hub.load("intel-isl/MiDaS", "MiDaS_small")
model.eval()

# Load specific preprocessing transforms for the MiDaS "small" variant
transforms = torch.hub.load("intel-isl/MiDaS", "transforms")
transform = transforms.small_transform

app = FastAPI()

# Enable CORS for frontend communication (crucial for local development)
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Serve the 'outputs' folder as a static directory so the frontend can access images and meshes via URL
app.mount("/outputs", StaticFiles(directory="outputs"), name="outputs")

class GenerateRequest(BaseModel):
    prompt: str
    negative_prompt: str = ""

def get_depth(image_path):
    """
    Predicts a depth map from a given image using the MiDaS model.
    Includes preprocessing, inference, and post-filtering.
    """
    img = cv2.imread(image_path)
    if img is None:
        raise Exception("Failed to load image for depth estimation")

    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    # Preprocess the image according to model requirements
    input_batch = transform(img)

    # Add batch dimension if necessary (NCHW format)
    if len(input_batch.shape) == 3:
        input_batch = input_batch.unsqueeze(0)

    with torch.no_grad():
        prediction = model(input_batch)
        depth = prediction.squeeze().cpu().numpy()

    # Normalize depth for visualization and filtering
    depth_vis = cv2.normalize(depth, None, 0, 255, cv2.NORM_MINMAX).astype("uint8")

    # Apply bilateral filter to smooth out noise while preserving edges
    depth = cv2.bilateralFilter(depth, 9, 75, 75)

    return depth

def depth_to_mesh(depth):
    """
    Converts a 2D depth map into a 3D point cloud and subsequently into a mesh using Poisson reconstruction.
    """
    h, w = depth.shape
    points = []
    for y in range(h):
        for x in range(w):
            z = float(depth[y][x])
            points.append([x, y, z])

    points = np.array(points)

    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(points)

    # Estimate normals for the point cloud, which is required for Poisson reconstruction
    pcd.estimate_normals(
        search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=5, max_nn=30)
    )

    # Generate the 3D triangle mesh from the points
    mesh, _ = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(pcd, depth=8)
    return mesh

def show_mesh(path):
    """ Utility function to open an Open3D visualization window for local debugging. """
    mesh = o3d.io.read_triangle_mesh(path)
    o3d.visualization.draw_geometries([mesh])

def save_depth_image(depth, path):
    """ Normalizes the float depth map to 8-bit and saves it as a PNG. """
    depth_vis = cv2.normalize(depth, None, 0, 255, cv2.NORM_MINMAX)
    depth_vis = depth_vis.astype("uint8")
    cv2.imwrite(path, depth_vis)

def run_triposr(image_path, run_id, suffix=""):
    """
    External call to the TripoSR local installation via subprocess.
    This generates a 3D OBJ mesh from a single 2D image.
    """
    local_output_dir = os.path.join("outputs", run_id)
    os.makedirs(local_output_dir, exist_ok=True)

    filename = os.path.basename(image_path)
    
    # Copy source image to TripoSR directory so run.py can find it locally
    triposr_input_path = os.path.join(TRIPOSR_DIR, filename)
    shutil.copy(image_path, triposr_input_path)

    # Define a temporary output folder inside TripoSR
    triposr_temp_out = os.path.join(TRIPOSR_DIR, "output_temp_" + run_id)

    # Command to run TripoSR's inference script
    command = [
        sys.executable,
        "run.py",
        filename,
        "--output-dir", triposr_temp_out,
        "--model-save-format", "obj"
    ]

    print(f"🚀 Starting TripoSR inference ({suffix})...")
    subprocess.run(command, cwd=TRIPOSR_DIR)

    # Search for the generated mesh.obj in the temporary folder
    found_file = None
    for root, dirs, files in os.walk(triposr_temp_out):
        for f in files:
            if f.endswith(".obj"):
                found_file = os.path.join(root, f)
                break
        if found_file: break

    if not found_file:
        raise Exception(f"TripoSR failed to generate a mesh for {suffix}")

    # Move the result to our project's output directory and rename it uniquely
    new_filename = f"{suffix}_{run_id}.obj"
    final_path = os.path.join(local_output_dir, new_filename)
    
    shutil.move(found_file, final_path)
    
    # Cleanup: remove temp folders and the copied input image
    shutil.rmtree(triposr_temp_out, ignore_errors=True)
    if os.path.exists(triposr_input_path):
        os.remove(triposr_input_path)

    print(f"✅ Mesh successfully moved to: {final_path}")
    
    # Return relative path for web access
    return f"outputs/{run_id}/{new_filename}"

async def generate_mesh_logic(image_path: str, run_id: str):
    """
    Main asynchronous pipeline: 
    1. Generates mesh from original image.
    2. Generates depth map.
    3. Generates mesh from depth map.
    """
    try:
        mesh_tasks[run_id] = {"status": "processing"}
        current_outputs_dir = os.path.join("outputs", run_id)
        os.makedirs(current_outputs_dir, exist_ok=True)

        # Step 1: Create mesh from the original image
        rel_mesh_original = run_triposr(image_path, run_id, "original")

        # Step 2: Extract Depth Map
        depth = get_depth(image_path)
        depth_filename = f"depth_{run_id}.png"
        depth_path = os.path.join(current_outputs_dir, depth_filename)
        save_depth_image(depth, depth_path)

        # Step 3: Create mesh specifically from the depth map visualization
        rel_mesh_depth = run_triposr(depth_path, run_id, "depth")

        # Save results in the task dictionary
        mesh_tasks[run_id] = {
            "status": "completed",
            "original_image_url": f"/outputs/{run_id}/{os.path.basename(image_path)}",
            "depth_image_url": f"/outputs/{run_id}/{depth_filename}",
            "original_mesh_url": f"/{rel_mesh_original}",
            "depth_mesh_url": f"/{rel_mesh_depth}"
        }
        print(f"✅ Task {run_id} completed successfully.")

    except Exception as e:
        print(f"❌ Error in generate_mesh_logic: {e}")
        mesh_tasks[run_id] = {"status": "error", "error": str(e)}    

@app.post("/generate")
def generate(req: GenerateRequest):
    """ Calls the Stable Diffusion API (Automatic1111) to generate an image from text. """
    print(f"Generating image - Prompt: {req.prompt}")
    
    url = "http://127.0.0.1:7860/sdapi/v1/txt2img"
    payload = {
        "prompt": req.prompt,
        "negative_prompt": req.negative_prompt,
        "steps": 10
    }

    try:
        response = requests.post(url, json=payload)

        if response.status_code != 200:
            return {
                "status": "error",
                "message": "Stable Diffusion API error",
                "status_code": response.status_code,
                "text": response.text
            }

        data = response.json()
        if "images" not in data:
            return {
                "status": "error",
                "message": "No images received from Stable Diffusion",
                "raw": data
            }
        return {"images": data["images"]}

    except Exception as e:
        return {"status": "error", "error": str(e)}

@app.post("/mesh")
async def create_mesh(request: Request, background_tasks: BackgroundTasks):
    """ Receives a Base64 image from the frontend and triggers the async 3D pipeline. """
    run_id = uuid.uuid4().hex
    data = await request.json()
    image_data = data.get("image")

    # Split Base64 header and data
    header, encoded = image_data.split(",", 1)
    image_bytes = base64.b64decode(encoded)

    # Setup unique output folder for this request
    current_outputs_dir = os.path.join("outputs", run_id)
    os.makedirs(current_outputs_dir, exist_ok=True)

    # Save original image as PNG
    original_filename = f"input_{run_id}.png"
    original_path = os.path.join(current_outputs_dir, original_filename)
    with open(original_path, "wb") as f:
        f.write(image_bytes)

    print(f"📸 Original image saved: {original_path}")
    
    # Start the CPU/GPU intensive mesh logic in the background
    background_tasks.add_task(generate_mesh_logic, original_path, run_id)
    return {"status": "ok", "run_id": run_id}

@app.post("/generate-texture")
async def generate_texture(request: Request):
    """
    Uses img2img via Stable Diffusion to transform the original image into 
    a flat, seamless albedo texture map.
    """
    try:
        data = await request.json()
        image_data = data.get("image")
        
        if "," in image_data:
            encoded = image_data.split(",")[1]
        else:
            encoded = image_data

        tex_id = uuid.uuid4().hex
        tex_dir = os.path.join("outputs", "textures", tex_id)
        os.makedirs(tex_dir, exist_ok=True)

        # Request to SD for texture creation
        sd_url = "http://127.0.0.1:7860/sdapi/v1/img2img"
        payload = {
            "init_images": [encoded],
            "prompt": "flat texture, albedo map, (seamless:1.3), high resolution, neutral lighting, centered, no shadows",
            "negative_prompt": "shadows, 3d, perspective, depth, dark edges, vignette, blur, glossy",
            "steps": 20,
            "cfg_scale": 7,
            "denoising_strength": 0.5, 
            "width": 512,
            "height": 512,
            "sampler_name": "Euler a"
        }

        print(f"🚀 Sending img2img request for texture generation...")
        response = requests.post(sd_url, json=payload, timeout=120)
        response.raise_for_status()
        
        data_sd = response.json()
        texture_base64 = data_sd["images"][0]
        texture_bytes = base64.b64decode(texture_base64)

        texture_filename = f"texture_{tex_id}.png"
        texture_path = os.path.join(tex_dir, texture_filename)
        
        with open(texture_path, "wb") as f:
            f.write(texture_bytes)

        print(f"✅ Texture generated and saved: {texture_path}")

        return {
            "status": "ok",
            "texture_url": f"/outputs/textures/{tex_id}/{texture_filename}"
        }

    except Exception as e:
        import traceback
        traceback.print_exc()
        return {"status": "error", "message": str(e)}

@app.post("/upload")
async def upload_image(background_tasks: BackgroundTasks, image: UploadFile = File(...)):
    """ 
    Handles direct file uploads from the user. 
    Includes security checks for file type and image integrity.
    """
    print(f"Processing uploaded file: {image.filename}")
    extension = image.filename.split(".")[-1].lower()
    if extension not in ALLOWED_EXTENSIONS:
        raise HTTPException(status_code=400, detail="File type not allowed!")

    contents = await image.read()
    print(f"Received data size: {len(contents)} bytes")
    
    # Image integrity check using Pillow
    try:
        img = Image.open(io.BytesIO(contents))
        img.verify() # Validates that the file is indeed an image
        print(f"Verified image format: {img.format}")
    except Exception:
        raise HTTPException(status_code=400, detail="Invalid image file detected!")
    
    run_id = uuid.uuid4().hex
    current_outputs_dir = os.path.join("outputs", run_id)
    os.makedirs(current_outputs_dir, exist_ok=True)

    unique_filename = f"{run_id}.{extension}"
    file_path = os.path.join(current_outputs_dir, unique_filename)

    # Store the uploaded file
    with open(file_path, "wb") as f:
        f.write(contents)

    background_tasks.add_task(generate_mesh_logic, file_path, run_id)
    return {"status": "ok", "run_id": run_id}

@app.get("/status/{run_id}")
async def get_status(run_id: str):
    """ Endpoint for the frontend to poll the current state of a mesh generation task. """
    return mesh_tasks.get(run_id, {"status": "not_found"})

main.py Herunterladen

letztes Update: 03.04.2026

Important: This documentation is currently a work in progress and may be incomplete.Suggestions for improvement are welcome and can be submitted via the contact form.

Deutsch:

1. Grundvoraussetzungen installieren

2. XAMPP (Frontend Hosting)

3. Stable Diffusion (AUTOMATIC1111)

Installation

Start-Konfiguration

Stable Diffusion starten

4. TripoSR (Image → 3D Mesh)

Installation

Virtuelle Umgebung

Dependencies

5. OpenMeshy Backend (FastAPI)

Installation

6. Projektstruktur

7. Open Meshy vorbereiten

8. Backend starten

9. Aufrufen der Webseite im Browser

Important: This documentation is currently a work in progress and may be incomplete.Suggestions for improvement are welcome and can be submitted via the contact form.

English:

1. Install Basic Requirements

2. XAMPP (Frontend Hosting)

3. Stable Diffusion (AUTOMATIC1111)

Installation

Startup Configuration

Start Stable Diffusion

4. TripoSR (Image → 3D Mesh)

Installation

Virtual Environment

Dependencies

5. OpenMeshy Backend (FastAPI)

Installation

6. Project Structure

7. Prepare Open Meshy

8. Start the Backend

9. Open the Website in Your Browser

Important: This documentation is currently a work in progress and may be incomplete.
Suggestions for improvement are welcome and can be submitted via the contact form.

Important: This documentation is currently a work in progress and may be incomplete.
Suggestions for improvement are welcome and can be submitted via the contact form.