Spring Cloud Data Flow Server for Cloud Foundry

Authors

Sabby Anandan, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Thomas Risberg, Marius Bogoevici, Josh Long, Michael Minella, David Turanski

1.2.3.BUILD-SNAPSHOT

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.


Table of Contents

I. Getting started
1. Deploying on Cloud Foundry
1.1. Provision a Redis service instance on Cloud Foundry
1.2. Provision a Rabbit service instance on Cloud Foundry
1.3. Provision a MySQL service instance on Cloud Foundry
1.4. Download the Spring Cloud Data Flow Server and Shell apps
1.5. Running the Server
1.5.1. Deploying and Running the Server app on Cloud Foundry
Configuring Defaults for Deployed Apps
1.5.2. Running the Server app locally
1.5.3. Sample Manifest Template
1.6. Running Spring Cloud Data Flow Shell locally
2. Application Names and Prefixes
2.1. Using Custom Routes
3. Deploying Docker Applications
4. Application Level Service Bindings
5. A Note About User Provided Services
6. Application Rolling Upgrades
7. Maximum Disk Quota Configuration
7.1. PCF’s Operations Manager Configuration
7.2. Scale Application
7.3. Configuring target free disk percentage
II. Applications
III. Architecture
8. Introduction
9. Microservice Architectural Style
9.1. Comparison to other Platform architectures
10. Streaming Applications
10.1. Imperative Programming Model
10.2. Functional Programming Model
11. Streams
11.1. Topologies
11.2. Concurrency
11.3. Partitioning
11.4. Message Delivery Guarantees
12. Analytics
13. Task Applications
14. Data Flow Server
14.1. Endpoints
14.2. Customization
14.3. Security
15. Runtime
15.1. Fault Tolerance
15.2. Resource Management
15.3. Scaling at runtime
15.4. Application Versioning
IV. Server Configuration
16. Feature Toggles
17. Database Configuration
17.1. Adding a custom JDBC driver
18. Security
18.1. Enabling HTTPS
18.1.1. Using Self-Signed Certificates
18.1.2. Self-Signed Certificates and the Shell
18.2. Traditional Authentication
18.2.1. Single User Authentication
18.2.2. File based authentication
18.2.3. LDAP Authentication
LDAP Transport Security
18.2.4. Shell Authentication
18.2.5. Customizing authorization
18.2.6. Authorization - Shell and Dashboard Behavior
18.2.7. Authorization with Ldap
18.3. OAuth 2.0
18.3.1. OAuth REST Endpoint Authorization
18.3.2. OAuth Authentication using the Spring Cloud Data Flow Shell
18.3.3. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
18.4. Securing the Spring Boot Management Endpoints
19. Monitoring and Management
19.1. Spring Boot Admin
19.2. Monitoring Deployed Applications
19.3. Log and DataDog MetricWriter
V. Shell
20. Shell Options
21. Listing available commands
22. Tab Completion
23. White space and quote rules
23.1. Quotes and Escaping
23.1.1. Shell rules
23.1.2. DSL parsing rules
23.1.3. SpEL syntax and SpEL literals
23.1.4. Putting it all together
VI. Streams
24. Introduction
24.1. Stream Pipeline DSL
24.2. Application properties
25. Lifecycle of Streams
25.1. Register a Stream App
25.1.1. Whitelisting application properties
25.1.2. Creating and using a dedicated metadata artifact
25.1.3. Using the companion artifact
25.2. Creating custom applications
25.3. Creating a Stream
25.3.1. Application properties
Passing application properties when creating a stream
25.3.2. Deployment properties
Application properties versus Deployer properties
Passing instance count as deployment property
Inline vs file reference properties
Passing application properties when deploying a stream
Passing Spring Cloud Stream properties for the application
Passing per-binding producer consumer properties
Passing stream partition properties during stream deployment
Passing application content type properties
Overriding application properties during stream deployment
25.3.3. Common application properties
25.4. Destroying a Stream
25.5. Deploying and Undeploying Streams
26. Stream DSL
26.1. Tap a Stream
26.2. Using Labels in a Stream
26.3. Named Destinations
26.4. Fan-in and Fan-out
27. Stream applications with multiple binder configurations
28. Examples
28.1. Simple Stream Processing
28.2. Stateful Stream Processing
28.3. Other Source and Sink Application Types
VII. Tasks
29. Introduction
30. The Lifecycle of a Task
30.1. Creating a Task Application
30.1.1. Task Database Configuration
30.2. Registering a Task Application
30.3. Creating a Task Definition
30.4. Launching a Task
30.4.1. Common application properties
30.5. Reviewing Task Executions
30.6. Destroying a Task Definition
31. Subscribing to Task/Batch Events
32. Launching Tasks from a Stream
32.1. TriggerTask
32.2. TaskLaunchRequest-transform
33. Composed Tasks
33.1. Configuring the Composed Task Runner
33.1.1. Registering the Composed Task Runner
33.1.2. Configuring the Composed Task Runner
33.2. The Lifecycle of a Composed Task
33.2.1. Creating a Composed Task
Task Application Parameters
33.2.2. Launching a Composed Task
Exit Statuses
33.2.3. Destroying a Composed Task
33.2.4. Stopping a Composed Task
33.2.5. Restarting a Composed Task
34. Composed Tasks DSL
34.1. Conditional Execution
34.2. Transitional Execution
34.2.1. Basic Transition
34.2.2. Transition With a Wildcard
34.2.3. Transition With a Following Conditional Execution
34.3. Split Execution
34.3.1. Split Containing Conditional Execution
VIII. Tasks on Cloud Foundry
35. Version Compatibility
36. Tooling
37. Task Database Schema
38. Running Task Applications
38.1. Create a Task
38.2. Launch a Task
38.3. View Task Logs
38.4. List Tasks
38.5. List Task Executions
38.6. Destroy a Task
38.7. Deleting Task From Cloud Foundry
IX. Dashboard
39. Introduction
40. Apps
40.1. Bulk Import of Applications
41. Runtime
42. Streams
43. Create Stream
44. Tasks
44.1. Apps
44.1.1. Create a Task Definition from a selected Task App
44.1.2. View Task App Details
44.2. Definitions
44.2.1. Creating Task Definitions using the bulk define interface
44.2.2. Creating Composed Task Definitions
44.2.3. Launching Tasks
44.3. Executions
45. Jobs
45.1. List job executions
45.1.1. Job execution details
45.1.2. Step execution details
45.1.3. Step Execution Progress
46. Analytics
X. REST API Guide
XI. Appendices
A. Data Flow Template
A.1. Using the Data Flow Template
B. Spring XD to SCDF
B.1. Terminology Changes
B.2. Modules to Applications
B.2.1. Custom Applications
B.2.2. Application Registration
B.2.3. Application Properties
B.3. Message Bus to Binders
B.3.1. Message Bus
B.3.2. Binders
B.3.3. Named Channels
B.3.4. Directed Graphs
B.4. Batch to Tasks
B.5. Shell/DSL Commands
B.6. REST-API
B.7. UI / Flo
B.8. Architecture Components
B.8.1. ZooKeeper
B.8.2. RDBMS
B.8.3. Redis
B.8.4. Cluster Topology
B.9. Central Configuration
B.10. Distribution
B.11. Hadoop Distribution Compatibility
B.12. YARN Deployment
B.13. Use Case Comparison
B.13.1. Use Case #1
B.13.2. Use Case #2
B.13.3. Use Case #3
C. Building
C.1. Documentation
C.2. Working with the code
C.2.1. Importing into eclipse with m2eclipse
C.2.2. Importing into eclipse without m2eclipse
D. Contributing
D.1. Sign the Contributor License Agreement
D.2. Code Conventions and Housekeeping