| 1 | issuekey | title | description | storypoint |
|---|
| 2 | XD-1 | HDFS ItemWriter | Base integration of core HDFS writer functionality with Spring Batch. | 1 |
| 3 | XD-2 | HDFS Core writing helper classes | Simple file writer that has existed in the spring hadoop samples. | 1 |
| 4 | XD-6 | Channel Registry | NULL | 3 |
| 5 | XD-7 | Tuple data structure | The tuple data structure should be backward compatible in functionality for use in spring batch. Porting over FieldSet tests in spring batch to use the tuple data structure is one way help ensure that compatibility. | 1 |
| 6 | XD-8 | Syslog Ingestion | Have a syslog.xml config file that can be added to a module and registered with a module registry. | 1 |
| 7 | XD-9 | Basic implementation of a reactor based tcp server | NULL | 13 |
| 8 | XD-10 | Reactor based http ingestion | When there is support for boostrapping a http server in the reactor project, and inbound SI adapter and associated XD source module should be created. | 5 |
| 9 | XD-11 | Initial work that uses the module architecture from DIRT | NULL | 5 |
| 10 | XD-12 | Jolokia based aggregator for cluster monitoring | NULL | 8 |
| 11 | XD-13 | Tail file channel adapters | NULL | 8 |
| 12 | XD-14 | Design for deploying XD on EC2 | create enough of a design to develop additional stories. | 3 |
| 13 | XD-18 | Websocket based taps | NULL | 5 |
| 14 | XD-20 | DIRT Runtime that deploys an application context across multiple nodes using redis. | NULL | 3 |
| 15 | XD-23 | add file source and sink modules | NULL | 1 |
| 16 | XD-24 | Create pipes and filters DSL for ingestion | Initial simple handcoded implementation for straight through pipe and filter model, e.g. a | b | c | 2 |
| 17 | XD-26 | Basic Performance Test For syslog injestion | NULL | 20 |
| 18 | XD-27 | Gemfire CQ module for ingestion | NULL | 1 |
| 19 | XD-28 | Create simple gague service | A gauge just stores a number. Implementations for in-memory and redis. | 1 |
| 20 | XD-29 | Create rich gauge service | A rich gauge stores a number and also rmd, min, max. Implementations for in-memory and redis. | 5 |
| 21 | XD-30 | Create a simple counter service | A simple counters can increment/decrement a number. Implementations for in-memory and redis. | 1 |
| 22 | XD-31 | Create field-value counters | A field-value counter is useful for bar chart graphs, Strings on x-axis and count on y-axis. Maps well to zset in redis. Implementations for in-memory and redis. | 5 |
| 23 | XD-35 | Create CI process for XD build | bamboo based | 1 |
| 24 | XD-36 | SI Outbound HDFS Channel Adapter | NULL | 3 |
| 25 | XD-37 | Gradle based multi-project build | multi project build. - look to Spring Framework for source of starting point. | 2 |
| 26 | XD-40 | Build script that creates an executable server as an artifact | Gradle application plugin is a good starting point. this should be the main server that would host SI based modules to do syslog->file ingestion (as an example) | 2 |
| 27 | XD-41 | Add JUnit @Rule so Tests Fail Fast with Clear Messaging if Redis Not Available | NULL | 1 |
| 28 | XD-43 | Metric repositories should support Spring Data CrudRepository interface | This provides common CRUD behavior and a shared interface that can be useful in testing scenarios. | 5 |
| 29 | XD-44 | Redis based repositories should use a NamingStrategy class to calculate the name of the key to use for persistence | RedisCounterRepository and RedisGaugeRepository have duplicated code that needs to be factored out into a one place. One such duplication is the determination of the key name to use for persistence. This should be abstracted out into a strategy helper class. | 1 |
| 30 | XD-45 | Remove the expiry of keys in Redis based repositories | There is duplicated code in Redis based repositories that related to expiry behavior, move into a common shared helper class and/or base class. | 1 |
| 31 | XD-47 | HDFS sink module | NULL | 1 |
| 32 | XD-48 | Parameterizable streams | NULL | 2 |
| 33 | XD-49 | Move Redis Queue Channel Adapters into spring-integration-redis | Currently these implementations are in the spring-xd-dirt module, but they should be moved into spring-integration-redis. We are already depending upon Spring Integration 3.0 snapshots since the ChannelRegistry implementation is not yet at a milestone, so this should be okay for the Redis Channel Adapters also - until Spring Integration M2 is released. | 3 |
| 34 | XD-50 | Add tap support to DIRT | syntax: {code} tap @ somechannel --key=value | somecounter {code} | 2 |
| 35 | XD-51 | Add xd.stream.name property in StreamPlugin | NULL | 1 |
| 36 | XD-52 | add twitter search source module | NULL | 1 |
| 37 | XD-53 | Design and document desired high level DSL for configuring data processing in XD | Start to explore how the DSL can cover both advanced (non-linear) spring integration flows as well as spring batch jobs. | 13 |
| 38 | XD-54 | XD Metrics backed Message Counter | A Spring Integration based @ServiceActivator that counts the number of messages using the Spring XD metrics support | 1 |
| 39 | XD-55 | SI ServiceActivator for an XD Metrics backed Field Value Counter | A Spring Integration based @ServiceActivator that counts the occurrence of field names, from either a tuple data structure or a POJO, using the Spring XD metrics support. | 3 |
| 40 | XD-56 | Switch to use Lettuce driver for Redis | Replace the use of Jedis with Lettuce as it has higher performance | 2 |
| 41 | XD-57 | add counter module | NULL | 1 |
| 42 | XD-58 | build.gradle doesn't handle a small handful of libraries | Trying to build spring-xd for the first resulted in lots of errors inside STS (I had an empty .m2 repo). | 1 |
| 43 | XD-59 | Tuple should support storing nested tuples | Nested tuple structures shoudl be supported, getTuple(int index), getTuple(String name) | 1 |
| 44 | XD-60 | Saving a metric (Counter, Gauge..) with an existing name should throw an exception | The difference between saving a new metric and updating an existing one needs to be defined. Suggest that if we try to save when an existing counter is already in the database to throw exception, such as DataIntegrityViolationException. | 1 |
| 45 | XD-61 | Create distributable artifact that contains server application and start/stop scripts | The gradle application task should get us most of the way to create a distributable artifact akin to what you see when downloading tomcat/jetty etc. Now there is a launch task task(launch, dependsOn: 'classes', type: JavaExec) { main = 'org.springframework.xd.dirt.stream.StreamServer' classpath = sourceSets.test.runtimeClasspath } The same main should be referenced in the application plugin, a task to create a .zip distributable is needed. Ideally would be nice to 1. download .zip 2. unzip 3. cd spring-xd/bin 4. xdserver start and gracefully shutdown later with 5. xdserver stop I don't know if we can/should bundle redis, I think we should bundle it. The scripts can be for unix/linux and for windows. Discuss a brew based install as well. | 8 |
| 46 | XD-62 | Use the tuple data structure to process data in a spring batch step | Do not require a POJO in order to do end-to-end processing in a batch step. | 5 |
| 47 | XD-63 | Document tuple data structure on XD wiki | NULL | 2 |
| 48 | XD-65 | Gemfire Sink to update a gemfire cache. | Update a gemfire region. | 2 |
| 49 | XD-66 | Gauge and Counter hash and equals should not depend on values | The value is mutable so this causes problems if storing metrics in a HashSet, for example. | 1 |
| 50 | XD-67 | Submit a brew-based install for Spring XD | - Host the Spring XD distributable zip somewhere that is accessible by external http request. - Create brew formula for Spring XD install while specifying redis as dependency. - starting up stream server upon successful brew install couple of questions: - should we name the brew task springxd? (name not taken yet) - should we start the stream server as part of the brew install process? - should we specify redis as a recommended dependency? user can pass in 'brew install springxd --without-redis' to skip redis installation. by default, 'brew install springxd' will install redis as well. | 8 |
| 51 | XD-68 | Export of data from HDFS to a relational database | Based on a single process running a Spring Batch job, support the ETL of data from HDFS to a RDBMS | 8 |
| 52 | XD-69 | Export of data from HDFS to MongoDB | Based on a single process Spring Batch job, ETL of data from HDFS to MongoDB. | 5 |
| 53 | XD-70 | Create general structure for AsciiDoc based wiki and Spring XD guide. | Adopt Asciidoc as the markdown syntax, useful for generating pdf and more feature rich than standard github flavored markdown. Loosely following the conventions of https://community.jboss.org/wiki/TheHolyGrailAsciiDocOnGitHubToDocBookTrain that have generate docbook/pdf docs from the Asciidoc wiki. The asciidoctor project is a key element in the adoption of AsciiDoc for use as the format in github, it is the rendering engine used by github for AsciiDoc. See http://asciidoctor.org/docs/asciidoc-writers-guide/ for guidance. | 2 |
| 54 | XD-71 | Remove UUID from Tuple class or replace with more efficient implementation | The Java UUID class is known not to be the fasted implementation available. See https://github.com/stephenc/eaio-uuid and http://mvnrepository.com/artifact/com.eaio.uuid/uuid for high perf impls. | 1 |
| 55 | XD-72 | Provide a http source | stream should be able to ingest data from http | 5 |
| 56 | XD-73 | add test to start/stop stream server | in additional to existing tests that check for redis connection, we need to add tests that start/stop stream server. | 2 |
| 57 | XD-74 | Create XD module for tail file adapter | NULL | 1 |
| 58 | XD-75 | Add redis bundle to distribution zip file | for linux and mac | 5 |
| 59 | XD-76 | Add gemfire-server application to the distribution zip of the project spring-xd-gemfire-server | NULL | 1 |
| 60 | XD-78 | Documentation for "http | file" processing | Put on the guide as a section in an 'input sources' wiki page. https://github.com/springsource/spring-xd/wiki/GuideGettingStarted | 1 |
| 61 | XD-79 | End user guide for data streams | Put on the guide as a section in an 'streams' wiki page. End user focused, no need to mention spring underpinning, impl details. | 5 |
| 62 | XD-80 | Documentation for "http | hdfs" processing | Put on the guide as a section in an 'input-stream' wiki page. | 1 |
| 63 | XD-81 | Documentation for "syslog | file" processing | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 64 | XD-82 | Documentation for "tail | file" processing | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 65 | XD-83 | Documentation for "tail | hdfs" processing | Put on the guide as a section in an 'input-stream' wiki page. | 1 |
| 66 | XD-84 | Documentation for "http | gemfire" processing | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 67 | XD-85 | Documentation for "gemfirecq | file" processing | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 68 | XD-86 | Documentation for "twittersearch | file" processing | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 69 | XD-87 | Documentation that introduces taps | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 70 | XD-88 | Documentation for counter taps | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 71 | XD-89 | Documentation for gauge taps | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 72 | XD-90 | Documentation for rich gauge taps | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 73 | XD-91 | Documentation for field value taps | Put on the guide as a section in an 'input-stream' wiki page. | 3 |
| 74 | XD-93 | Normalize and refactor component packaging decomposition | Normalize and refactor as needed functionality currently included in - spring-integration-core (LocalChannelRegistry) - spring-integration-module (Module types upon which Flow are built) - xd-module (Depend Module types common to DIRT and Spring Integration) - spring-integration-flow (Flow specific types, namespace support, etc) | 5 |
| 75 | XD-94 | Document The ability to use flows in streams | Test and document e.g. "http | flow1 | flow2 | file" | 3 |
| 76 | XD-95 | The ability to override default module path for the plugin or an individual flow | Currently modules are assumed to be in classpath:/META-INF/spring/integration/module/${flow}.xml. To reuse modules defined with DIRT may require more flexibility. e.g., <int-flow:flow root-path="file:///dirt/module"/> | 3 |
| 77 | XD-96 | The ability to import beans referenced in the main context into a module | This should be a core feature of any Spring based module. The plugin should be able to import explicitly referenced beans. This minimizes potential side effects of making the module a child context and is simpler than declaring a shared context (parent) of the application and the module. Provide namespace support for flow: | 3 |
| 78 | XD-97 | The ability to route within streams | A Flow or a processor component may require routing semantics. Currently the stream assumes a single input and output for each module. A Flow may support multiple outputs - Switch routing that is - Recipient list is not currently supported (another subtask?). We need to support semantics like: a |[output.foo:c,output.bar:d,default:e] | 8 |
| 79 | XD-98 | Documentation on the module system and how to contribute new modules | For people who are familiar with Spring/Spring Integration provide documents that show how to add additional input sources/sinks. | 3 |
| 80 | XD-99 | Sonar build is failing | https://build.springsource.org/browse/XD-SONAR-34 Caused by: java.lang.ClassNotFoundException: org.sonar.api.Plugin at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50) at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:244) at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:230) ... 94 more | 1 |
| 81 | XD-100 | Rename Tuple class in spring-xd-tuple | The Tuple classes in Reactor follow the more traditional data structure concept of Tuples, an immutable fixed length sequence of values where each value can have different types. They are ordered and can often be access by index. An example in a static language is the Tuple class found in .NET http://msdn.microsoft.com/en-us/library/system.tuple.aspx or in Scala http://www.tutorialspoint.com/scala/scala_tuples.htm Using this standard definition of a Tuple, they do not support named values. There is also a different tuple class instance for each length, e.g. Tuple<T1,T2>, Tuple<T1,T2,T3>. The Tuple class in XD is more like a record or named tuple. Python has a named tuple concept - http://docs.python.org/2/library/collections.html#collections.namedtuple and http://stackoverflow.com/questions/1490413/languages-that-allow-named-tuples shows that other languages use the term 'Record' for a 'named tuple' - Haskell, Standard ML, OCaml, and F#. http://en.wikibooks.org/wiki/F_Sharp_Programming/Tuples_and_Records#Defining_Records So boiling it all down, to avoid conflicts of names, and also to open up the possibility of using Reactor tuples as keys (instead of strings for names), we should change the name to either NamedTuple or Record. ATM, there is no direct relationship between Reactor's Tuple and NamedTuple (such as inheritance) and so probably Record is the way to go. | 2 |
| 82 | XD-101 | Add gradle tasks that build and bundle the redis server | NULL | 2 |
| 83 | XD-102 | Create XDContainer class to start stream server | Provide optional command line arg to embed the container launcher, aka - xd-admin server. XDContainer.sh --embeddAdmin | 1 |
| 84 | XD-103 | Create XDAdmin server to start container launcher | This will launch the RedisContainerLauncher, in future will be able to select from a variety of middleware options. | 1 |
| 85 | XD-104 | Add README to be included in root directory of distribution | should explain basic layout of the distribution | 1 |
| 86 | XD-105 | Add LICENSE to be included in root directory of distribution | should contain apache licence | 1 |
| 87 | XD-106 | Container server does not log a message that it has started or stopped successfully | $ ./xd-container processing module 'Module [name=file, type=sink]' from group 'tailtest' with index: 1 processing module 'Module [name=tail, type=source]' from group 'tailtest' with index: 0 Logging of 'processing module' should have log level, time.. | 1 |
| 88 | XD-107 | Clean shutdown of redis in xd-admin | A ctrl-c of xd-admin results in exception messages about disconnecting from redis. 14:16:07,327 ERROR task-scheduler-1 handler.LoggingHandler:136 - org.springframework.data.redis.RedisSystemException: Redis command interrupted; nested exception is com.lambdaworks.redis.RedisCommandInterruptedException: Command interrupted | 2 |
| 89 | XD-108 | Build script should not package 'spring-xd-dirt' scripts | We are packaging separate scripts to start XDAdmin and XDContainer. The Gradle application plugin will generate an unwanted 'spring-xd-dirt' scripts, this should be removed from the bin directory when creating a distribution zip. | 1 |
| 90 | XD-109 | Documentation for starting Spring XD servers | NULL | 1 |
| 91 | XD-110 | Remove use of system property xd.home to define location for install location, rely on environment variable XD_HOME | This is in the AdminMain and ContainerMain. Can get the environment property directly in java code unless provided explicitly on the command line using --xdHomeDir. | 1 |
| 92 | XD-111 | Create final distribution zip across multiple projects | The final directory structure should look like <install-dir>/xd <install-dir>/redis <install-dir>/gemfire inside the XD directory /xd/bin - which has xd-container and xd-admin scripts /xd/lib inside the gemfire directory /gemfire/bin - has the gemfire-server script /gemfire/lib inside the redis directory is /redis/redis-latest-v.x.y.z.tar /redis/README /readis/install-redis - script that does the basic 4 commands to install redis. There should be a gradle task that runs after the distZip task, that will take the contents of different project directories, script diretories and 'redis-binary' directories and creates the final layout for the distribution. | 5 |
| 93 | XD-112 | Update XD to Use SI 3.0.0.M2 | NULL | 1 |
| 94 | XD-114 | Add install script for Redis | This assumes the redis source tar is available under $rootDir/redis/redis-2.6.13.tar.gz The install script does the following: - Check the platform OS & arch - unzip the tar, compile the sources | 2 |
| 95 | XD-115 | add SpEL 'transform' processor | It should provide an 'expression' param for SpEL and have a default pass-thru of the payload. | 1 |
| 96 | XD-116 | add SpEL 'filter' processor | It should provide an 'expression' param for SpEL and have a default value of true (accept everything). | 1 |
| 97 | XD-117 | add spring-integration-groovy to container dependencies | This will enable the use of groovy scripts within modules. | 1 |
| 98 | XD-118 | replace testsource with time and testsink with log | This should facilitate testing while avoiding any class dependencies. Also, log is a generally useful sink by itself and time is a more interesting source for testing (should accept --interval for the seconds between time messages). | 1 |
| 99 | XD-119 | HDFS sink should default to hdfs://localhost:8020 | The current default is hdfs://localhost:9000 but most new distributions/installs use 8020 | 1 |
| 100 | XD-120 | Find and eliminate package-level cycles across XD projects | NULL | 3 |
| 101 | XD-121 | Add pre-compiled redis distributions for the selective OS platforms | NULL | 10 |
| 102 | XD-122 | XD scripts need to have spring-integration milestone versions updated | Spring-integration version is changed to 3.0.0.M2 and since we manually create the XD scripts, they still point to the 3.0.0.BUILD-SNAPSHOT version. As discussed, we also need to have a better strategy on updating the lib directory inside the XD scripts. | 2 |
| 103 | XD-123 | XD scripts lib path needs to be dynamic | We currently have the manually created XD scripts. This makes it difficult to maintain as the lib path is error prone with the changes. We need to make sure that the properties such as lib path etc., are dynamically updated. | 3 |
| 104 | XD-124 | Clean shutdown of redis in xd-container | Need to shutdown cleanly, no exception messages are shown. Order of components in the stream should be shut down from 'first to last' (opposite of creation) | 2 |
| 105 | XD-125 | Stream documentation review | NULL | 2 |
| 106 | XD-126 | Documentation for sources, sinks, modules should define which attributes are required and which optional | This will eventually be supplied by the admin server, but for now write it up by hand in the documentation | 2 |
| 107 | XD-127 | Create TCP source module | Based off SI tcp inbound adapter. This will allow for event forwarding that can select among the existing SI serialized/deserializer options. | 3 |
| 108 | XD-128 | Create TCP sink module | Based off SI tcp inbound adapter. This will allow for event fowarding. | 3 |
| 109 | XD-129 | Documenation for building/starting redis servers | NULL | 2 |
| 110 | XD-130 | Remove container entry in Redis when the application context event to shutdown the container is fired | NULL | 3 |
| 111 | XD-131 | Upgrade Lettuce to 2.3.2 | NULL | 2 |
| 112 | XD-132 | Profile support for modules | To allow for groups of beans to be defined or not in the container that runs a module. When deploying a stream (e.g. via the REST API), it should be possible to also provide profile names. Then those would apply to any modules within that particular stream deployment. | 8 |
| 113 | XD-133 | Fail Sonar CI build if there are any package tangles violated. | Similar to what would show up on structure101 reports. | 1 |
| 114 | XD-134 | Investigate link checking tool for user guide | Asciidoc/doctor might have one as part of it toolchain | 2 |
| 115 | XD-135 | Documentation on XD Architecture | Show overall flow of data in a stream, the server components 'admin' and 'container'. How modules are deployed. | 4 |
| 116 | XD-136 | Documentation that points on how to install hadoop | Pointers to other documentation on how to install hadoop. | 3 |
| 117 | XD-137 | Release Spring XD 1.0 M1 | NULL | 5 |
| 118 | XD-138 | Prepare blog post for M1 | NULL | 5 |
| 119 | XD-139 | Update README.txt to include instructions on how to build | Building XD should not be part of the out first out of the box experience, but we should include some instructions on what targets are available, such as distXD. | 2 |
| 120 | XD-140 | Parameterize syslog Source; Add Support for TCP | The syslog source currently is hard-coded to use udp on port 11111. Need to parameterize the port and provide an option to use TCP. | 2 |
| 121 | XD-141 | install-redis script should not use relative path to determine redis source dist | Currently, the install-redis script uses relative path to determine redis source dist file. Since this is error prone, we need to fix it. | 1 |
| 122 | XD-142 | StreamServer Context Lifecycle Issues | The {{ModuleDeployer}} calls {{getBeansOfType}} before the context has had its {{PropertySourcesPlaceholderConfigurer}} attached. This can cause issues with {{FactoryBean}} s with placeholders in constructor args because the unresolved placeholder is used when the {{FactoryBean}} is pre-instantiated to determine the type of object it will serve up. | 8 |
| 123 | XD-143 | Create externalized property file to support connectivity to redis | We need to have an externalized property file(under xd/conf/) for the xd-container & admin scripts to use as options. | 3 |
| 124 | XD-147 | Remove use of application plugin for redis project | Currently redis project uses application plugin to bundle distribution. This also includes 'java plugin' which causes java specific build behavior on this project. We should try removing the use of application plugin and use something similar or custom tasks that does the bundling. | 2 |
| 125 | XD-148 | Improve User Experience when Redis is not running | Redis is not running we get a nasty stacktrace: {code} ~/dev/git/spring-xd/dist/spring-xd/xd/bin (master)] ➔ ./xd-container 13/05/29 16:17:15 INFO support.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@851052d: startup date [Wed May 29 16:17:15 EDT 2013]; root of context hierarchy 13/05/29 16:17:16 INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/spring/launcher.xml] 13/05/29 16:17:16 INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [META-INF/spring/redis.xml] 13/05/29 16:17:17 INFO support.DefaultListableBeanFactory: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4934ce4a: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,redisConnectionFactory,org.springframework.xd.dirt.launcher.RedisContainerLauncher#0]; root of factory hierarchy 13/05/29 16:17:17 INFO support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4934ce4a: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,redisConnectionFactory,org.springframework.xd.dirt.launcher.RedisContainerLauncher#0]; root of factory hierarchy Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisConnectionFactory' defined in class path resource [META-INF/spring/redis.xml]: Invocation of init method failed; nested exception is com.lambdaworks.redis.RedisException: Unable to connect org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1488) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461) org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292) org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626) org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83) org.springframework.xd.dirt.launcher.RedisContainerLauncher.main(RedisContainerLauncher.java:68) org.springframework.xd.ContainerMain.main(ContainerMain.java:68) Caused by: com.lambdaworks.redis.RedisException: Unable to connect com.lambdaworks.redis.RedisClient.connect(RedisClient.java:176) com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:139) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.initConnection(LettuceConnectionFactory.java:108) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.afterPropertiesSet(LettuceConnectionFactory.java:86) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1547) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485) ... 13 more Caused by: java.net.ConnectException: Connection refused: localhost/127.0.0.1:6379 sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) {code} I think it would be helpful to provide users with some helpful advice e.g.: Redis does not seem to be running at 'localhost/127.0.0.1:6379'. Did you start or install Redis? Please see for further help http://foo/bar | 2 |
| 126 | XD-149 | Create asciidoc toolchain script to create a 'toc2' style html output | Publishing an html version of the guide that uses the 'toc2' style format, table of contents on the left. Looks like a 'stylesheet factory' http://asciidoctor.org/docs/produce-custom-themes-using-asciidoctor-stylesheet-factory/ needs to be installed. From the theme showcase, http://themes.asciidoctor.org/preview/, the 'golo' theme has a toc2 style. In the root of the git repo for the wiki is a build.gradle file that uses the asciidoctor gradle plugin, but it doesn't support using a single file with import as input. (See See https://github.com/asciidoctor/asciidoctor-gradle-plugin/issues/15 ) The mvn plugin does support this, so maybe using mvn is an option or just a bash script. | 4 |
| 127 | XD-150 | Publish Spring XD final distribution zip as part of Bamboo artifactory plugin | Currently, Bamboo's gradle artifactory plugin has the artifacts configured to projects target(build) directory 'archives'. We need to have a way to set the final distribution archive as one of the gradle 'configurations' in our build.gradle and refer it inside bamboo artifacts. | 2 |
| 128 | XD-151 | Add Redis binaries for Windows | Presently, Spring XD does not ship Windows binaries for Redis. However, Microsoft is actively working [1] on supporting Redis on Windows. You can download Windows Redis binaries from: https://github.com/MSOpenTech/redis/tree/2.6/bin/release [1] http://blogs.msdn.com/b/interoperability/archive/2013/04/22/redis-on-windows-stable-and-reliable.aspx | 2 |
| 129 | XD-152 | Create rich gauge module | Spring config for rich gauge plus message handler to coerce a numeric or string payload to a double. | 5 |
| 130 | XD-153 | create a gauge module | Spring config for simple gauge plus message handler to process message. There is some code common to RichGaugeHandler to coerce the payload to a double that should be refactored for reuse. | 5 |
| 131 | XD-154 | Provided console output of started server | Shouldn't we have something like a ContextRefreshedEvent Listener and output some informational messages to the console, so the user knows the Container is up (Which contexts. Maybe even print a link to the docs))? Maybe even some simple ascii art (for demos)? Right now it looks somewhat barren. Redis provides something similar. This may even go hand in hand to provided a better configuration model (storing common config parameters centrally) {code} _____ _ __ _______ / ____| (_) \ \ / / __ \ | (___ _ __ _ __ _ _ __ __ _ \ V /| | | | \___ \| '_ \| '__| | '_ \ / _` | > < | | | | ____) | |_) | | | | | | | (_| | / . \| |__| | |_____/| .__/|_| |_|_| |_|\__, | /_/ \_\_____/ | | __/ | |_| v1.0.0.M1 |___/ eXtreme Data Using Redis at localhost:6379 The Server (PID: 12345) is now ready on http://myserver:123/streams Documentation: https://github.com/SpringSource/spring-xd/wiki {code} | 2 |
| 132 | XD-155 | Add a groovy script processor module | A processor module that accepts either the location of a groovy script resource or an inline script (string). Also some discussion about a default classpath location for scripts. | 5 |
| 133 | XD-156 | Create config support for Redis | We would like to have Redis driven from a config property file under XD_HOME. | 2 |
| 134 | XD-157 | Verify use of JMX managed bean to shutdown cleanly the xd-admin and xd-container servers | In both cases there are multiple application contexts that can be running in the process. The JMX Managed bean should call 'close' on those application contexts. The more detailed lifecycle of cleanly shutting down components within those application contexts is another story. | 4 |
| 135 | XD-158 | Expose shutdown operation over http | This will allow for a simple way to shutdown the server via an HTTP call. Support for security is a separate story. The end goal is to have some shell scripts distributed that can issue HTTP requests to shutdown the xd-admin and xd-container servers. The newest version of Jolokia has the ability to boostrap itself inside an application context vs. requiring a java agent. I suspect using the application context approach will provide us with more flexibility (e.g. property replacement etc) but not sure. | 4 |
| 136 | XD-159 | Parameter parsing does not work if an argument contains '--'. | Parameter parsing does not work if an argument contains '--'. For example: {code} ... | transform --expression=42 | transform --expression=--payload |... {code} Also, I was surprised that this worked.. {code} | transform --expression=new StringBuilder(payload).reverse() | {code} ... but this didn't... {code} | transform --expression='new StringBuilder(payload).reverse()' | {code} I think we need to tokenize the argument (with ' if contains spaces) and remove any surrounding '...' from the result. This means if someone wants a SpEL literal they would have to use something like {code}--expression=''Hello, world!''{code} resulting in a SpEL literal 'Hello, world!' | 1 |
| 137 | XD-160 | Publish golo themed docs documentation to static.springsource.org as part of nightly build | The wiki repo contains a script, gen-docs.sh, that we are planning to use to generate a pretty HTML version of the Getting Started guide. We should consider using maven (or gradle, but there is currently an issue documented in build.gradle) to generate this and other reference docs and publish them automatically as part of a nightly build. | 3 |
| 138 | XD-161 | Add HTTP Delete Stream Operation | NULL | 5 |
| 139 | XD-162 | Create design document for implementation strategy to support message conversion in ChannelRegistry | The conversion should be based on content-type headers, similar to the way Spring's HttpMessageConverters work (with mime types). Also, the map of available converters should be extensible while including the most common defaults (for JSON, XML, etc). We most likely want to add a few of our own content types also (e.g. for Tuples). Most likely, this logic and the configuration methods for extending the converter map, belong in AbstractChannelRegistry since it should be common across all implementations (i.e. the logic should be the same regardless of the transport used after-serialization/before-deserialization). | 5 |
| 140 | XD-163 | Enable grouping of modules for co-located deployment | example: {code} a | (b | c) | d {code} ...where b and c modules are deployed together as a composite module. There are 2 options (maybe more) for how we could handle that. One would be defining a CompositeModule type that simply bridges the channels (b's output to c's input in this example). The second option would be to deploy those together on the same node as modules but using the LocalChannelRegistry between them. | 12 |
| 141 | XD-164 | Validate processing modules declare the required channels | Validate that modules have required channels declared according to their type. Currently the stream deployer accepts processors with no input, but the stream doesn't complete. We should fail earlier and more loudly. | 2 |
| 142 | XD-165 | xd-container and xd-admin should log to a file out of the box | We should have an externally editable log4j config file in a conf dir and the default should log to a file (presumably in a logs dir) | 2 |
| 143 | XD-166 | Create config support based on channel registry type | We need to have the XD container & admin reading the registry specific property based on the registry type selected. From Mark F, on one of the code review comments: Maybe rather than having redis, rabbit, etc. properties all within a container.properties we should rely upon naming conventions instead. Specifically, we could have a single configurable property for the type of channel registry ("redis", "rabbit", or "local" being possible values), and then we could use something like: <import resource="config/${registry.type}.xml"/> <context:property-placeholder location="config/${registry.type}.properties"/> | 3 |
| 144 | XD-167 | Script to generate reference documentation from wiki and include in .zip distribution | NULL | 5 |
| 145 | XD-168 | Decide on location to host http reference documentation and automate upload in build scripts | NULL | 4 |
| 146 | XD-169 | XD should run offline | Trying to run XD offline results in an error in redis.xml because the cloudfoundry schema file is missing. We need to add the cf-runtime jar to the classpath to resolve this. | 2 |
| 147 | XD-170 | Home wiki page improvements | Add more structure, more easily find the reference guide. The style that is here https://github.com/snowplow/snowplow/wiki is nice. | 2 |
| 148 | XD-171 | Create project home page for SpringXD on springsource.org/spring-xd | A minimal project page of a 'top level project' page that has basic information of docs and links to the github wiki page. No need to list maven coordinates. | 4 |
| 149 | XD-172 | Create links to SpringXD on other pages of springsource.org site | bottom home page - list of projects data/integration category landing pages - related projects. | 4 |
| 150 | XD-173 | Document how to create a custom input/output module for existing SI channel adapters | Document how to take an existing input/output channel adapters in spring integration and add them as a XD source/sink module. Should be as end-user focused, step by step guide as possible. Consider including a getting started gradle/pom.xml | 6 |
| 151 | XD-174 | Document how to create a custom processor module. | The use case is to write custom code that does processing on a specific domain class (perhaps from twitter adapter) or a tuple. Need to package up this code so that it can be used inside XD. | 4 |
| 152 | XD-175 | Create and document a syslog aggregation example | 2-3 containers (separate processes) that the stream: syslog | tcp 1 container (separate process) that aggregates the data sent from those conainers, tcp | severityFilter | hdfs | 8 |
| 153 | XD-176 | Support exponential moving average in RichGauge | This could easily be supported in the existing gauge by adding a setAlpha method to RichGaugeService and adding the extra parameter "alpha" to the gauge data (https://en.wikipedia.org/wiki/Exponential_moving_average). If not set it would default to the current behaviour (simple mean), otherwise it would calculate the exponential moving average in place of the mean. | 2 |
| 154 | XD-177 | Easily switch between a single process that performs all admin and processing tasks to one that has a dedicated admin processes and distributed processing containers. | NULL | 13 |
| 155 | XD-178 | DefaultContainer should have a default constructor that generates a UUID | The current incrementAndGet approach based off redis will not easily be applicable in local model deployment | 1 |
| 156 | XD-179 | Have three startup scripts, xd-singlenode, xd-admin, and xd-container | The xd-singlenode script will launch a main application that creates both the admin node (to process http admin requests) and the container node (to execute modules for data processing) within in the same process the xd-admin script will launch a main application that creates only the admin node (remove current embeddedContainer options) the xd-container script will launch a main application that creates only the container node (as it is now) | 1 |
| 157 | XD-180 | The command line for xd-admin and xd-container to support an additional option, pipeProtocol, that is used to determine the middleware for sending admin requests and data between processing steps | The name 'pipeProtocol' is tentative. 1. The command line scripts for xd-admin and xd-container would support a --pipeProtocol option, with the default being to use Redis. (Otherwise use xd-singlenode). 2. The xd-admin and xd-container scripts will use the value of pipeProtocol to set the java system property xd.pipeProtocol when launching the app. | 1 |
| 158 | XD-181 | Update launcher.xml to have protocol independent beans defined and an import statement to load protocol specific defintiions from a system property defined location. | launcher.xml can make use of the system property xd.pipeProtocol inside an import statement. This determines which version of the XD infrastructure to load, for example what ChannelRegistry implementation, Local or Redis based, or specific message listener containers. File name conventions should be used, so if the option passed in from the command line is --pipeProtocol localChannel then the XML filename looked for has the 'Protocol' suffix applied, e.g. localChannelProtocol, and is loaded via the classpath. Redis and Local will not be the only options, other implementations will be provided in the future, e.g. Rabbit, and the user may be able to provide their own implementations of these infrastructure classes (an advanced task). | 3 |
| 159 | XD-182 | Create redisProtocol.xml that will load all the Redis specific implementations to suppor the XD container runtime and administration | The redis specific beans that are defined in the current launcher.xml should move into this configuration file. | 3 |
| 160 | XD-183 | Create localChannelProtocol.xml that will load all the SI specific implementations to suppor the XD container runtime and administration | NULL | 3 |
| 161 | XD-184 | Add unregistration support to the channel registry | NULL | 4 |
| 162 | XD-185 | Refactor StreamServer to an interface and create Redis and Local implementations | The current StreamServer depends on RedisStreamDeployer. Call this RedisStreamServer and extract interface to allow alternate implementations | 2 |
| 163 | XD-186 | Create a pipe protocol independent StreamDeployer | Create StreamDeployer that does not depend on an adapter implementation | 2 |
| 164 | XD-187 | Create XD script for xd-single node | This script will launch XD admin along with the module container. As part of this implementation, we will also remove the embedded options for XD admin & container scripts. | 2 |
| 165 | XD-188 | Add directory to classpath in server startup scripts so groovy based processors can be easily referenced by name without a resource uri prefix | a stream such as time | filter --script=oddMinuteFilter.groovy | file would load the groovy script 'oddMinuteFilter.groovy' that is located in the directory modules/processor or perhaps in modules/processor/scripts. Not sure the benefit of having a subdirectory below processor just for scripts. | 1 |
| 166 | XD-189 | Investigate running XD on Cloud Foundry | NULL | 5 |
| 167 | XD-190 | Cleanup embedded container story | The --embeddedX options are a bit confusing in code right now, as the Admin can embed the Container and vice-versa. I guess we should only keep the Admin>Container side of things. | 1 |
| 168 | XD-191 | Move Redis connection metadata logging into the code closest to establishing that connection | XD-106 included detailed logging about the Redis metadata within the RedisContainerListener, but it seems as though that info could be logged somewhere closer to the establishment of a Redis connection for the XD runtime (and could be logged even if this listener, whose main role is to capture Container-related events, is not enabled). | 2 |
| 169 | XD-192 | Update getting started documentation to use xd-singlenode start script. | With the new option of starting without requiring redis, the getting started documentation should reflect this easier way to start processing data. | 2 |
| 170 | XD-193 | Need more unique resource locations for XD internal configuration | Currently internal config files are in META-INF/spring with fairly generic names. To avoid potential collisions if users add their own configuration in the classpath, we should have a more unique location, e.g. META-INF/spring/xd | 3 |
| 171 | XD-194 | Users should be able to package custom modules into a single jar | If a user needs to deploy a module containing custom code, they have to build a jar for the lib dir and somehow deploy the app context file separately to a modules dir. This is a bit inconvenient to build from a project, since context files in src/main/resources typically get built into a jar. Might be better to accept both jar and xml files in the modules dir, though that brings up the issue of classpath isolation. | 5 |
| 172 | XD-195 | The {{time}} Source Should Emit String by Default | When running in local mode (no Redis) {{time | tcp}} no longer works. Change the {{time}} source to emit the date as a String, while allowing an option to emit a {{Date}} object. | 1 |
| 173 | XD-196 | replace the hacky parser with a good one | Replace the existing DSL parser that uses string indexing with a more robust one based on a derivative of SpEL. This will provide a stable base on which to quickly iterate on syntax. | 4 |
| 174 | XD-197 | Create a scriptProcessor module that allows the execution of a groovy (potentially jruby,jython) based SI Service Activator | This will enable arbitrary processing logic to be used in a processing step. See http://blog.springsource.org/2011/12/08/spring-integration-scripting-support-part-1/ <int:service-activator ...> <script:script lang="groovy" location="file:scripts/groovy/myscript.groovy"> </int:service-activator> would be the essence of the module. Probably 'lang' gets detected from the file extension. | 4 |
| 175 | XD-198 | Documentation for developing streams in the IDE needs to mention including scripts dir to project classpath | {{curl -X POST -d "time --interval=3 | transform | log" http://localhost:8080/streams/test}} results in the following stack trace in the DEBUG log. It's apparently benign, but ugly... {code} 2013-06-06 10:43:36,875 [task-scheduler-1] DEBUG: org.springframework.scripting.support.ResourceScriptSource - class path resource [transform.groovy] could not be resolved in the file system - current timestamp not available for script modification check java.io.FileNotFoundException: class path resource [transform.groovy] cannot be resolved to URL because it does not exist org.springframework.core.io.ClassPathResource.getURL(ClassPathResource.java:177) org.springframework.core.io.AbstractFileResolvingResource.lastModified(AbstractFileResolvingResource.java:170) org.springframework.scripting.support.ResourceScriptSource.retrieveLastModifiedTime(ResourceScriptSource.java:101) org.springframework.scripting.support.ResourceScriptSource.getScriptAsString(ResourceScriptSource.java:79) org.springframework.integration.scripting.RefreshableResourceScriptSource.<init>(RefreshableResourceScriptSource.java:46) sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) java.lang.reflect.Constructor.newInstance(Constructor.java:513) org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147) org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:121) org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:280) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1035) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:939) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:126) org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616) org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1035) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:939) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:271) org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:126) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:589) org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925) org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472) org.springframework.xd.module.SimpleModule.start(SimpleModule.java:97) org.springframework.xd.dirt.module.ModuleDeployer.deployModule(ModuleDeployer.java:120) org.springframework.xd.dirt.module.ModuleDeployer.handleMessageInternal(ModuleDeployer.java:108) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69) org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:84) org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:57) org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102) org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:102) org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:126) org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:230) org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:129) org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:73) org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:67) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:137) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:92) org.springframework.integration.x.redis.RedisQueueInboundChannelAdapter.access$4(RedisQueueInboundChannelAdapter.java:1) org.springframework.integration.x.redis.RedisQueueInboundChannelAdapter$ListenerTask.run(RedisQueueInboundChannelAdapter.java:110) org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) {code} | 1 |
| 176 | XD-199 | Ensure the DELETE Operation can Delete a Tap | While deleting a stream doesn't remove any taps right now, we should be able to explicitly delete a tap. Determine whether the current DELETE works and, if not, make it so. | 3 |
| 177 | XD-200 | Creating a tap throws an exception | Creating a tap throws an exception. In local mode: Cannot resolve reference to bean 'redisConnectionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'redisConnectionFactory' is defined But also fails when using redis. | 4 |
| 178 | XD-201 | Fix XD scripts on windows | Currently the XD scripts are broken in windows. | 2 |
| 179 | XD-202 | Update twittersearch module for Twitter 1.0 API retirement | NULL | 3 |
| 180 | XD-203 | Provided modules should be integration tested | I don't see that we have automated tests for the modules we provide out-of-the-box. We could make the modules folder an Eclipse project (which would also help solve XD-198) and add some integration tests similar to those documented here: https://github.com/SpringSource/spring-xd/wiki/Creating-Custom-Modules | 5 |
| 181 | XD-204 | Document processor modules | Fill in https://github.com/SpringSource/spring-xd/wiki/Processors | 2 |
| 182 | XD-205 | Processor modules should support scripts in languages other than groovy | Filter, Transform, and Script modules all assume the provided script is written in Groovy. This is partly due to the fact that the "lang" attribute of <int-script:script> can't be set to a property value (i.e. lang="${lang:groovy}"), which would allow users to pass in the expected language. Or perhaps we could use a SPEL expression or script to pick the language based on the file extension? | 3 |
| 183 | XD-206 | XD AdminMain & ContainerMain should check xd.home property from scripts | Currently, the system property xd.home is set as JVM_OPTS (via SPRING_XD_ADMIN_OPTS) into xd-admin & xd-container scripts. Inside the ContainerMain & AdminMain, we need to check if this system property is set and use it. It seems like, this check is missing now. | 1 |
| 184 | XD-207 | Provide configurable properties for hdfs sink. | The config file modules/sink/hdfs.xml has a hardcoded value to locate the namenode. <hdp:configuration register-url-handler="false"> fs.default.name=hdfs://localhost:9000 </hdp:configuration> the fs.default.name proprety should be configurable and we should also support loading an external configuration file using <hdp:configuration properties-location="${xd.home}/config/hadoop.properties"> </hdp:configuration> | 2 |
| 185 | XD-208 | Document the file sink | NULL | 1 |
| 186 | XD-209 | Create or document existing project template for custom module creation | The wiki docs on creating custom modules have entire build.gradle files in them. Would be good to explore existing STS templates, maven archetypes, etc or create new ones for authoring custom modules. | 3 |
| 187 | XD-210 | If output directory does not exist for a file sink, by default allow it to be created | There shouldn't be a need to do a mkdir -p before sending data to a file sink. | 1 |
| 188 | XD-211 | Document the log sink | NULL | 1 |
| 189 | XD-212 | Add http port command line option to AdminMain | Currently StreamServer has setPort, but no way for end user to set it. | 2 |
| 190 | XD-213 | Temporarily add toString() Logic in Local Mode Inter-Module Comms | NULL | 2 |
| 191 | XD-214 | Create documentation on the general DSL syntax | The asciidoc wiki should have a section (included in the _Sidebar.asciidoc as well) that describes the general usage of the DSL syntax. | 3 |
| 192 | XD-215 | Add authentication information to twittersearch source doc | Since the changes for XD-202, twittersearch requires authentication. Need to update the docs to reflect this. | 1 |
| 193 | XD-216 | Add support for tap foo.bar syntax in the DSL | NULL | 2 |
| 194 | XD-217 | Add config dir to classpath to support custom properties-locations | The transform, filter, and script processor modules support passing in a properties-location for script variables. We need a default location on the classpath for users to provide custom properties files. | 2 |
| 195 | XD-218 | Add support to load a twitter.properties file in the source | NULL | 1 |
| 196 | XD-219 | Surpress tap WARNING message in local mode | NULL | 1 |
| 197 | XD-220 | Add twitter oauth properties file to config dir | Those property keys should then be provided as defaults for the placeholders in source/twittersearch.xml | 2 |
| 198 | XD-221 | Links in asciidoctor generated HTML+docbook documentation are broken | The issue arises because the link:document[Label] asciidoc macro is meant for "external documents" and creates {{<ulink>}} in docbook / {{<a href="document">}} in html, whereas we want {{<link linkend="anchor">}} / {{<a href="doc#anchor>}} resp. We also want it to continue working in github live view. I guess what could work is to have the macro (either override the link macro or create our own if github supports that) that looks like : {{link:document#anchor[Label]}} (the #anchor works out of the box in asciidoc and should work in github) but override it for the html and docbook backends to render to the correct form. The thing is, there are several ways to create/override macros (and templates they render to), some of which make sense to our setup: - having asciidoc.conf in the directory of the processed document (http://asciidoc.org/userguide.html#X27) - having docbook.conf/html.conf in the directory of the processed document (http://asciidoc.org/userguide.html#X27) - defining macros using attributes (http://asciidoc.org/userguide.html#_setting_configuration_entries) I tried all of those, but to no avail. These DO WORK with plain asciidoc, but not with our toolchain. Don't know if the problem is with asciidocTOR or with the gradle wrapper though. | 2 |
| 199 | XD-222 | Add docs for Deleting a simple stream. | curl -X DELETE http://localhost:8080/streams/ticktock | 1 |
| 200 | XD-223 | Update Creating a Custom Source Module doc with a different SI adapter due to Twitter issues | https://github.com/SpringSource/spring-xd/wiki/Creating-a-Source-Module uses the SI twittersearch inbound channel adapter, which is no longer going to work once Twitter disallows anonymous searches. Ideally we update the example to use a new version of SI-twitter that adds support for this (as opposed to the XD workaround.) | 2 |
| 201 | XD-224 | Reduce necessity for quoting in parameter values in DSL expressions | parameter values that include spaces need to be quoted, but this becomes overly complex in this kind of case. Here is what you want to say: {code} http --port=9995 | filter --expression=payload.matches('hello world') {code} With the rule 'parameter values that contain spaces must be quoted' it would be this: {code} http --port=9995 | filter --expression='payload.matches('hello world')' {code} But then to include single quotes within a single quoted string you need to use two of them: '' - so it becomes {code} http --port=9995 | filter --expression='payload.matches(''hello world'')' {code} Less than ideal. | 2 |
| 202 | XD-225 | field-value-counter should support nested fieldNames | for example, when using the 'twittersearch' source module, the "hashTags" are nested within "entities", and the value for hashTags is itself an object with a "text" field, so the following would be needed to count the actual value of interest: {code} tap @ tweets | field-value-counter --fieldName=entities.hashTags.text {code} | 2 |
| 203 | XD-226 | Cleanup and Optimize gradle tasks to bundle spring-xd distribution | We need to cleanup some of the duplicate gradle tasks that bundle spring-xd distributions. Currently, distXD does the copy of distributions from "spring-xd-dirt", "redis" and "spring-xd-gemfire-server" projects into "$rootDir/dist/spring-xd". And, the task "zipXD" makes the zip archive. These tasks should be combined with the "distZip" & "docZip" tasks. We also need to remove the duplicate artifacts configuration from these tasks. | 2 |
| 204 | XD-227 | Add jetty-util-6.1.26.jar and jsr311-api-1.1.1.jar as required jars so they will be on the XD classpath | This is needed for the use of the webhdfs:// scheme to talk to HDFS over http. | 1 |
| 205 | XD-228 | Missing '=' in example of http stream | In documentation attached to M1, in Streams/Introduction section, there's {noformat} http --port 8091 | file --dir=/tmp/httpdata/ {noformat} while it should be: {noformat} http --port=8091 | file --dir=/tmp/httpdata/ {noformat} missing "{{=}}" in {{http}} | 1 |
| 206 | XD-229 | Add RabbitMQ-based implementation of ChannelRegistry | NULL | 8 |
| 207 | XD-230 | Add RabbitMQ source module | configurable parameters should include the queue-name(s) and optional binding key pattern connection info, such as host and port, should also be configurable but with defaults (localhost and default port), and that should likely fallback to a rabbit.properties file in the $XD_HOME/config directory | 3 |
| 208 | XD-231 | Add Twitter gardenhose source module | we have a prototype gardenhose adapter that was built directly upon RestTemplate (streaming on a background thread), but Spring Social Twitter has an issue on its 1.1 roadmap that is relevant: https://jira.springsource.org/browse/SOCIALTW-2 | 3 |
| 209 | XD-232 | A HATEOAS designed REST API using Spring HATEOAS library. | NULL | 21 |
| 210 | XD-233 | Create an shell to control all aspect of stream/job management | NULL | 21 |
| 211 | XD-234 | Support various output format, e.g. Avro, SequenceFile, more advanced rollover options. | NULL | 21 |
| 212 | XD-235 | Modules (sinks, processors, sources) should be able to be easily tested inside the IDE using JUnit | NULL | 21 |
| 213 | XD-236 | Create an Aggregate Counter | An aggregate counter rolls up counts into discrete time buckets. There is an existing POC implementation in Java based off the library https://github.com/thheller/timed-counter The README there has a good description of the desired feature set. | 8 |
| 214 | XD-237 | Spring Batch jobs should be able to be deployed to the DIRT runtime and managed. | NULL | 21 |
| 215 | XD-238 | Deploying Custom Code | When a module is deployed, it should run in its own isolated classpath. The current code has all dependencies in a single classpath, taken from the lib directory at startup. This has a number of drawbacks, one of the most important is the batch jobs can not be contributed to the system at runtime. The work for this epic is decoupled from any module deployment story. The assumption is that there will be a directory layout as shown below. Current layout ./modules/. |-- common |-- job |-- processor |-- sink |-- source |-- trigger And inside source |-- source | |-- file.xml | |-- gemfire-cq.xml | |-- gemfire.xml | |-- http.xml | |-- jms.xml | |-- mqtt.xml | |-- rabbit.xml | |-- syslog-tcp.xml | |-- syslog-udp.xml | |-- tail.xml | |-- tap.xml | |-- tcp.xml | |-- time.xml | |-- twittersearch.xml | |-- twitterstream.xml Using an example of the source directory from the current layout.e.g ./modules/source/file, the new layout would be ./modules/source/file/lib/spring-integration-file.jar ./modules/source/file/config/file.xml We should support both the new and old layout styles simultaneously. There what is under 'file' directory is the 'package' No .zip, war, is required. | 21 |
| 216 | XD-240 | Throughput optimized TCP based adapters based on Reactor TCP project | NULL | 21 |
| 217 | XD-242 | Add JMS source module | NULL | 2 |
| 218 | XD-244 | Create a Trigger | h2. Narrative As the XD system, I need to be able to execute a job (or potentially a stream) based on a given condition (time, data existence, etc). This story is intended is for a local trigger implementation but remote triggers will also need to exist. h2. Acceptance Criteria # Implement the ability to register a time based trigger {{trigger <CRON_STRING>}} for example # Implement the ability to register a file existence based trigger {{trigger <PATH>}} for example # Implement the ability to execute a job via an anonymous trigger: {{job_name @ <CRON_STRING OR PATH>}} # Implement the ability to execute a job via a job via the previously registered trigger: {{job_name @ trigger_name}} | 8 |
| 219 | XD-245 | Deploy Batch Jobs on XD | h2. Narrative As a developer, I need a way to deploy job configurations as well as the related custom code to XD. h2. Acceptance Criteria # Provide the ability to register jobs that have been deployed as modules via something like {{curl -d "job" http://localhost:8080/streams/myJob}} where job is the name of the job definition located in /modules/job and myJob is the name of the resulting registered job # Confirm that both "regular" jobs and Spring Hadoop based jobs can be packaged/run. | 8 |
| 220 | XD-246 | MessageChannelItemWriter | h2. Narrative As a user of XD, I want to be able to use a job as a source. To do so, we need the output of a job to be written to a message channel h2. Acceptance Criteria # Create a new ItemWriter in the Spring Batch project to write to a Spring Integration message channel. | 4 |
| 221 | XD-247 | Need to be able to specify password for Redis | Running on Cloud Foundry (and other managed environments) we need to be able to specify a Redis password in addition to host and port. | 2 |
| 222 | XD-250 | Set up a project for XD REST client library | NULL | 5 |
| 223 | XD-251 | Add support for stream creation | For stream creation we need to be able to specify: source sink processor - filter - transformer - script etc. | 10 |
| 224 | XD-255 | Set up a project for XD Shell | Set up a basic Spring Shell project for XD Shell | 3 |
| 225 | XD-256 | Create a banner page for XD Shell | NULL | 2 |
| 226 | XD-257 | Create the base implementation for XDCommands for the shell | This is the basic setup of the commands file - no specific command implementations | 3 |
| 227 | XD-258 | Add command for stream creation | NULL | 3 |
| 228 | XD-259 | Add command for tap creation | NULL | 1 |
| 229 | XD-260 | Add command for listing streams | NULL | 3 |
| 230 | XD-261 | Add command for deleting a stream | NULL | 1 |
| 231 | XD-262 | Convert current REST servlet to Spring MVC | NULL | 3 |
| 232 | XD-263 | Support GET /streams | Pagination support, maybe querying by name as well | 5 |
| 233 | XD-264 | Spring MVC infrastructure tests | NULL | 8 |
| 234 | XD-265 | Retrieve description of all registered modules | GET /streams/{streamname}/modules and GET /streams/{streamname}/modules/{modulename} The former returning links to the latter | 5 |
| 235 | XD-266 | Create java client lib over REST API | So that clients (e.g. Shell or custom user program) are insulated from REST details (ala Cloud Foundry). May go even further if we want a Java DSL for stream definitions (that may reuse Batch command POJOs btw): Difference between: xdClient.createStream("mystream", "http --port=9000 | file") and import static stuff.*; StreamDef stream = http().port(9000).pipe(file()); xdClient.createStream("mystream", stream); | 8 |
| 236 | XD-267 | Job as a Source | h2. Narrative As an XD developer, I need to be able to use a batch job to stream data as a source. h2. Acceptance Criteria # Implement the ability for a job to be defined as a source in the DSL # Add the configurations for the batch infrastructure transparently to the user # Add the ability to specify if the job is stateful (picks up where it left off if it stops or restarts at the beginning). | 10 |
| 237 | XD-268 | Streamline command-line arg management | Command line arguments (and especially their default values) are currently scattered around different places. The aim is to regroup those in a common place (*Options classes make sense). Also, not very happy with how System properties are used as a vehicle for options.transport / options.home | 2 |
| 238 | XD-269 | Create JobDefinition repository | h2. Narrative As XD, I need a persistent way to register job definitions (beyond the map registry implementation provided by Spring Batch). h2. Acceptance Criteria # XD should be able to register, unregister, and find job definitions via the registry. # The registry should be backed by Redis so that it is persistent. | 4 |
| 239 | XD-270 | The HDFS Sink should support a file naming strategy to distinguish between file currently being written and completed files | A file that is in the process of being written to should have a customized suffix added to the name, e.g. 'temp'. Once the file is closed, the suffix is removed and replaced with another value - default value can be dependent on the serialization format used, but can be customized | 8 |
| 240 | XD-271 | The HDFS Sink should support a number of rollover options | A strategy to roll over files that allows the user to choose between 1) the size of the file 2) the number of events/items in the file 3) an idle timeout value that if exceeded, will close the file | 8 |
| 241 | XD-282 | The HDFS Sink should support writing POJOs to HDFS using Avro Serialization | Writing POJOs using CDK Data (Avro) We should support both partitioned and un-partitioned. This story addresses only un-partitioned. Document limitations in terms of which Java types are supported and not supported by the Avro serialization | 10 |
| 242 | XD-290 | Redis backed container's RedisQueueInboundChannelAdapter is not performant | Currently, the RedisQueueInboundChannelAdapter has blocking operation when pulling the messages out of redis queue and this is not performant. There are few ideas from the discussion to make it better: 1) Get more items from the redis queue per connection 2) We will also have compression of messages(at the channel registry) before being sent to the redis queue We also need to investigate what redis connection strategy makes the RedisQueueInboundAdapter better. | 2 |
| 243 | XD-291 | HTTP Source still listens on port 9000 after removal. | Steps to reproduce: 1. curl -d "http | log" http://localhost:8080/streams/testHttp 2. curl -X DELETE http://localhost:8080/streams/testHttp 3. curl -d "http | log" http://localhost:8080/streams/testHttp org.jboss.netty.channel.ChannelException: Failed to bind to: 0.0.0.0/0.0.0.0:9000 | 1 |
| 244 | XD-292 | Redis 'install-redis' script fails on Ubuntu64 | The installation script for redis fails on Ubuntu64 when trying to untar the redis distribution. The script uses REDIS_ZIPNAME instead of REDIS_ZIP_PATH. This bug will be seen on any Linux 64 bits platform and looking at the code, even Linux 32 bits platform. | 1 |
| 245 | XD-293 | Ensure package-info.java is present for each package | We should ensure that each Package of Spring XD is documented. Right now the created JavaDoc looks barren: http://static.springsource.org/spring-xd/docs/1.0.0.M1/api/ | 4 |
| 246 | XD-294 | Apply JavaDoc HotFix | NULL | 1 |
| 247 | XD-295 | redis.properties values ignored | The container application loads {{redis.properties}}, but for some reason the values are ignored, and defaults are used instead. Repro steps: # Unpack Spring XD 1.0.0.M1 to a machine with no running Redis instance # Change /xd/config/redis.properties to specify a different hostname # Run /xd/bin/xd-container # Observe error about inability to connect to Redis on localhost Workaround * Pass -Dredis.hostname={desired IP} as a JVM parameter | 1 |
| 248 | XD-296 | Add log config file to gemfire in final distro | The changes for XD-144 mean that log4j files are no longer in the library jars. The admin server already has a logging configuration which should be activated by the startup scripts, but the separate gemfire app doesn't. | 1 |
| 249 | XD-297 | File sink should support rollover | I wanted to have a rollover feature when I was streaming tweets to a file overnight, just to avoid dealing with a single enormous file (in case I collected more data than my demo could handle and needed to split it up). | 2 |
| 250 | XD-298 | Refactor gardenhose into more generic twitterstream source | Twitter's streaming APIs have more capabilities than just the plain statuses/sample.json. In particular we should support the filter.json option and the use of "track" (https://dev.twitter.com/docs/streaming-apis/parameters#track) as well as other request parameters (delimited, language etc). | 2 |
| 251 | XD-299 | Creating a tap with same name as existing streams results in infinite loop | See http://stackoverflow.com/questions/17157068/counter-analytics-in-springxd The underlying issue is stream creation with a name already taken though | 5 |
| 252 | XD-301 | Request to create a repo for Spring XD performance testing | It would be nice if we have a git repo for Spring XD performance testing. This would enable us to have a common repository (rather than inside spring-xd as a subproject) for all performance related code specific to any module, message middleware etc., | 4 |
| 253 | XD-302 | User wants ability to create a mock source | To send a pre-set message to process(es) | 8 |
| 254 | XD-303 | User wants ability to create a in-process sink or tap | So that we can validate the message content in the stream | 8 |
| 255 | XD-304 | User wants ability to test processors | Be able to point to the processor xml file, e.g. modules/processors/transformer.xml, and have access to a source channel that drives messages into the processor and a output channel where output messages are send. The outbound channel is queue backed. Test sending JSON to a processor module that uses Tuples. | 8 |
| 256 | XD-305 | User wants ability to test sinks | Handled by 1245 | 8 |
| 257 | XD-306 | User wants ability to test sources | Examples: 1. Be able to start the rabbitmq source just by pointing to modules/source/rabbit.xml, pass in some property file for parameters to be replaced, and outgoing message is placed in a in-memory queue backed channel for use with assertions to verify functionality. 2. Test for as many source types as is 'reasonable', e.g. MQTT/TCP testing might be harder than say rabbitmq. 3. Test that sending json, results in media-type header is set to json 4. Test that sending POJO, " POJO 5. Test that sending Tuple, " Tuple 6. Test that sending raw bytes, " raw bytes | 8 |
| 258 | XD-307 | User to send a message directly to module and receive a message from a module | NULL | 8 |
| 259 | XD-308 | User wants ability to test multiple processors in a chain | NULL | 8 |
| 260 | XD-309 | Users should be able to package custom modules as a single zip file | See XD-194 for additional considerations. Zip support should be similar to uber-jar, or possibly replace uber-jar support. | 8 |
| 261 | XD-312 | Add Jolokia Agent Depending on Run Mode | WAR Vs. JVM Jolokia Agent Jolokia Vs. JVM MBeanServer Probably needs support for Spring Profiles. | 5 |
| 262 | XD-313 | Add Spring/Integration MBean Exporters to Module ApplicationContexts | Global option? Override for individual modules? module types? | 8 |
| 263 | XD-315 | Package Shell "binary" next to xd-admin and xd-container | The shell should be an 'executable' delivered out of the box in much the same way that xd-container and xd-admin are right now. If we follow how redis/mongo distribut the shell, it sits side by side with the other binaries | 3 |
| 264 | XD-316 | Create a common exception framework for XD | Need to capture exceptions from the various projects that make up XD and wrap them in XD Specific exceptions. An example of this is when leaving out the channels in the module definitions, we see NoSuchBeanExceptions and IllegalArgumentExceptions thrown based on which module and what channel is missing. | 5 |
| 265 | XD-317 | Add Documentation Chapter on Executing Batch Jobs | NULL | 4 |
| 266 | XD-318 | Create Splunk sink module | This would be based off the spring-integration-extenstions splunk project. The use of this adapter for storing tweet data is in https://github.com/markpollack/springone We should be able to reproduce the use case as done in that demo | 10 |
| 267 | XD-319 | Add command for deleting a tap | NULL | 1 |
| 268 | XD-320 | Support for GET of /taps | NULL | 1 |
| 269 | XD-321 | Add create() and deploy() to TapsController | POST? | 1 |
| 270 | XD-322 | Support for DELETE of taps | NULL | 1 |
| 271 | XD-323 | Add command for listing of taps | NULL | 1 |
| 272 | XD-328 | Retrieve information for a Counter | NULL | 5 |
| 273 | XD-329 | Retrieve information for a Field Value Counter | TODO as part of this (see XD-537): * Get rid of so-called Service layer in analytics project (doesn't do much right now, and logic would better live in the 'Handler' IMO) * Have REST controllers depend on XRepository in all cases | 3 |
| 274 | XD-330 | Retrieve information for a Gauge | TODO as part of this (see XD-537): * Get rid of so-called Service layer in analytics project (doesn't do much right now, and logic would better live in the 'Handler' IMO) * Have REST controllers depend on XRepository in all cases | 3 |
| 275 | XD-331 | Retrieve information for a Rich Gauge | TODO as part of this (see XD-537): * Get rid of so-called Service layer in analytics project (doesn't do much right now, and logic would better live in the 'Handler' IMO) * Have REST controllers depend on XRepository in all cases | 3 |
| 276 | XD-332 | Retrieve information for an aggregate counter | TODO as part of this (see XD-537): * Get rid of so-called Service layer in analytics project (doesn't do much right now, and logic would better live in the 'Handler' IMO) * Have REST controllers depend on XRepository in all cases | 3 |
| 277 | XD-334 | Document the structure of the REST API | NULL | 5 |
| 278 | XD-335 | Review DSL | updated story points to 14 since 5 of us just participated in a 2 hour call, and we still need to discuss "topology" support after some dev spikes later this week | 14 |
| 279 | XD-336 | Document Splunk source sink | NULL | 2 |
| 280 | XD-337 | Test connection pooling on Redis blocking/nonblocking operations | NULL | 2 |
| 281 | XD-338 | Create tcp/udp load generator script for XD performance testing | Create a load generator script which can generate messages at specific 1) Rate 2) Payload 3) Concurrency to a specific tcp/udp port where a syslog adapter is listening. | 4 |
| 282 | XD-339 | Investigate using profiler when doing the performance testing | Investigate how efficiently we can integrate profiler into the performance test. | 1 |
| 283 | XD-340 | Create script to extract table data from JSON based on a given HAWQ table structure | We should be able to write a script that can examine the table structure for a given HAWQ table and then extract the data from JSON without the custom script we are using now. | 8 |
| 284 | XD-341 | Document JMX features | Document jmx command line options and refer to jolokia | 2 |
| 285 | XD-342 | Fix classpath error caused by multiple conflicting servlet-api jars | There is some conflicting Servlet API jars on the claspath that needs cleanup. Building and running with xd-singlenode script gave this error: Jun 27, 2013 3:18:16 PM org.apache.coyote.http11.AbstractHttp11Processor process SEVERE: Error processing request java.lang.NoSuchMethodError: javax.servlet.ServletContext.getEffectiveSessionTrackingModes()Ljava/util/Set; org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:674) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:402) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) | 3 |
| 286 | XD-343 | Investigate JMX object naming of deployed modules and inbound/outbound channel adapters. | The object naming is still not ideal for XD since SI conventions add some noise. Likely need to design and implement a custom naming strategy | 5 |
| 287 | XD-344 | Add BatchMbeanExporter for batch modules | NULL | 5 |
| 288 | XD-345 | Replace "gardenhose" doc with new "twitterstream" | NULL | 1 |
| 289 | XD-346 | Esper based Complex Event Processing module | I would like to see a module created that supports complex event processing. I have reviewed GemFire Continuous Query but was not able to find a feature for time windows. I have used Esper in the past for this type of processing. | 1 |
| 290 | XD-347 | Investigate Redis connection timeout issues when running performance test | With the performance test run, the numbers (messages sent/received per second) keep varying as there are "redis client connection timeout exceptions" (Caused by: org.jboss.netty.channel.ConnectTimeoutException: connection timed out) at both redis inbound/outbound channel adapters as I increase the total number of messages being processed (max. 10K/second). Some of the exception messages for the review: 1) With connection pool (at Redis outbound): Caused by: org.springframework.data.redis.connection.PoolException: Could not get a resource from the pool; nested exception is com.lambdaworks.redis.RedisException: Unable to connect at org.springframework.data.redis.connection.lettuce.DefaultLettucePool.getResource(DefaultLettucePool.java:95) at org.springframework.data.redis.connection.lettuce.DefaultLettucePool.getResource(DefaultLettucePool.java:36) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.createLettuceConnector(LettuceConnectionFactory.java:318) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:109) at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:81) at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:53) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:157) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:137) at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84) at org.springframework.data.redis.core.DefaultListOperations.leftPush(DefaultListOperations.java:71) at org.springframework.data.redis.core.DefaultBoundListOperations.leftPush(DefaultBoundListOperations.java:67) at org.springframework.xd.perftest.redis.outbound.RedisQOutboundChannelAdapter.handleMessageInternal(RedisQOutboundChannelAdapter.java:71) at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) ... 17 more Caused by: com.lambdaworks.redis.RedisException: Unable to connect at com.lambdaworks.redis.RedisClient.connect(RedisClient.java:176) at com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:139) at org.springframework.data.redis.connection.lettuce.DefaultLettucePool$LettuceFactory.makeObject(DefaultLettucePool.java:252) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1181) at org.springframework.data.redis.connection.lettuce.DefaultLettucePool.getResource(DefaultLettucePool.java:93) ... 29 more Caused by: org.jboss.netty.channel.ConnectTimeoutException: connection timed out: localhost/127.0.0.1:6379 at org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137) at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) 2) Without connection pool (at Redis inbound): Caused by: com.lambdaworks.redis.RedisException: Unable to connect at com.lambdaworks.redis.RedisClient.connect(RedisClient.java:176) at com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:139) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.createLettuceConnector(LettuceConnectionFactory.java:321) ... 12 more Caused by: org.jboss.netty.channel.ConnectTimeoutException: connection timed out: localhost/127.0.0.1:6379 at org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137) at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ... 3 more | 2 |
| 291 | XD-348 | Trigger - Add support for fixed-delay interval | Trigger - Add support for fixed-delay interval | 1 |
| 292 | XD-349 | Trigger - Add support for date-based one-time execution | Currently Jobs can be either executed using cron expression or immediately at once. We should also support the one-time scheduling of jobs in the future. Would this possibly require us to implement schedule-persistence? That could severely impact story-points. | 4 |
| 293 | XD-350 | Support hourly resolution in redis aggregate counter | NULL | 4 |
| 294 | XD-351 | Support daily query resolution in redis aggregate counter | NULL | 4 |
| 295 | XD-352 | In-memory implementation of aggregate counter | NULL | 3 |
| 296 | XD-353 | Clean up Spring Configuration | - Container context should be separate from Admin Context in local mode (consistency across transports). Provide a LocalChannelRegistry to bridge deploy and undeploy channels to the ModuleDeployer - Verify Plugins are not in common module context. They are only needed by the ModuleDeployer not the Modules. - Add global-beans Config for beans to be shared among admin and container (and available to modules). This would be set as the parent context where needed. (Currently ‘analytics-context.xml’ and JobRepository shared by Admin and modules) - Fix ModuleDeployer sets parentContext twice - Rename common.xml to module-common.xml NOTE: Analytics parent only required in local mode - Write unit tests to verify configurations are as expected. Correct bean implementations and no stray beans or redundant instances where not needed - Decouple Command options from System properties (in general XD property names), this gives us some flexibility in mapping to properties or profiles where appropriate | 5 |
| 297 | XD-354 | Investigate Reactor-based Dispatchers in the common ApplicationContext that can be used by Modules | NULL | 8 |
| 298 | XD-355 | Enable configuration of Executors in source and sink modules, by default using Dispatchers in parent context | Need to understand how individual modules may or may not share Dispatchers that are part of the parent context. If modules have their own dispatchers, those also need to be configurable. | 8 |
| 299 | XD-356 | Test startup scripts on windows | startup scripts on windows should be tested, xd-admin, xd-container, xd-shell. | 1 |
| 300 | XD-357 | Creating a base class for Plugins | It might be worth creating a base class for Plugins that combines common concerns across plugins. E.g. That would allow us to hide the commonApplicationContext and BeanDefinitionAddingPostProcessor for common cases, instead exposing a simple addBeanDefinition method to sub-classes." | 4 |
| 301 | XD-358 | Add support for creating named cron triggers | Simple cron based triggers | 2 |
| 302 | XD-359 | Add support for creating a spring batch job that has an embedded trigger expression | NULL | 1 |
| 303 | XD-360 | Add support for creating a spring batch job that references a named trigger | NULL | 2 |
| 304 | XD-361 | Create a trigger from Shell | NULL | 1 |
| 305 | XD-363 | Add support for creating fixed delay/ fixed rate triggers | NULL | 1 |
| 306 | XD-364 | Create TriggerDefinition Repository | Redis based. | 1 |
| 307 | XD-365 | Support having multiple property placeholders defined in different modules | Right now the PPC for jolokia mgmt is conflicting with the PPC used to resolve redis properties. Need to determine a strategy such that multiple PPCs can be used. | 4 |
| 308 | XD-366 | Create a XD job definition | when posting the DSL to create a spring batch job e.g. "trigger job.xml --option1=foo" it should be stored (in redis) so that a listing of XD job definitions can be retrieved. | 3 |
| 309 | XD-367 | Final review of REST API structure document for streams, taps and jobs | Get closure on open discussion points for REST API wrt to streams, taps and jobs. | 3 |
| 310 | XD-368 | Improve connection handling in RedisAggregateCounterService. | This is currently too chatty. It should be possible to use a single connection for each "increment" operation. | 2 |
| 311 | XD-369 | Further DSL extensions | Extend the DSL in the following ways: - stream naming, use <name>'=' {code} mystream = http | file {code} - module aliasing (for later referencing) use <label>':' {code} mystream = http | t1: transform --expression=payload.toUpperCase() | t2: transform --expression=payload.toLowerCase() {code} - sequence of job steps with '&' {code} mybigjobby = step1 --option=value & step2 --option=value {code} - Channels for sources and sinks, use '>', channels references prefixed ':' {code} // sink channel called foo http | transform --expression=payload.toUppercase() > :foo // source channel called foo :foo > count | file {code} - Qualify channels with a stream ':'<stream>'.'<channel> {code} mystream = http | transform --expression=payload.toUppercase() > :foo :mystream.foo > count | log {code} - Reusable substreams, define then reuse {code} myIntricateFlow = transform --expression=payload.toUppercase() | transform --expression=payload.toLowercase() http | myIntricateFlow | file {code} - Parameterized substreams, use $XX or ${XX} to indicate parameters {code} obfuscateName = transform --expression=payload.replaceAll("${name}",XXX) twitter --query=Bieber | obfuscateName --name=Justin | file {code} - Tapping (still not 100% happy about the format here) {code} mystream = http | filter | t1: transform XXX | t2: transform YYY | file // These are then equivalent (tapping on a module is effectively tapping a channel, hence the '>') tap filter > count | file tap mystream.filter > count | file mystream = http | filter > :foo tap :foo > count | file tap :mystream.foo > count | file {code} I'd still like to thing about removing 'tap' and using a symbol (perhaps '@', but @:mystream.foo is a little odd) still not covered, topological constraints on the stream components. | 3 |
| 312 | XD-370 | Add command for deploying a Stream | Deploy a named stream. The stream must exist in the StreamRepository | 3 |
| 313 | XD-371 | Command for creating a job | optional --autostart switch to also deploy the job | 1 |
| 314 | XD-372 | Command to deploy a job | Deploy an existing job. Must exist in the JobsRepository | 1 |
| 315 | XD-373 | Command to create a tap | To store it's definition and optionally deploy with --autostart flag | 1 |
| 316 | XD-374 | Command to list taps | NULL | 1 |
| 317 | XD-375 | Command to delete tap | NULL | 1 |
| 318 | XD-376 | add create() and deploy() methods to JobsController | see https://docs.google.com/a/gopivotal.com/drawings/d/1kCNbVspRBjGc10itF9cSkwST8WgCjgg3wGzFKqLCAzU/edit Create the controller if it doesn't exist. Test with MvcTest | 1 |
| 319 | XD-377 | add create() and deploy() methods to JobDeployer | see https://docs.google.com/a/gopivotal.com/drawings/d/1kCNbVspRBjGc10itF9cSkwST8WgCjgg3wGzFKqLCAzU/edit Create the deployer if it doesn't exist. | 1 |
| 320 | XD-378 | add create() and deploy() methods to StreamsController | see https://docs.google.com/a/gopivotal.com/drawings/d/1kCNbVspRBjGc10itF9cSkwST8WgCjgg3wGzFKqLCAzU/edit create optionally deploys | 1 |
| 321 | XD-379 | add create() and deploy() methods to StreamDeployer | see https://docs.google.com/a/gopivotal.com/drawings/d/1kCNbVspRBjGc10itF9cSkwST8WgCjgg3wGzFKqLCAzU/edit Refactor current DefaultStreamDeployer | 1 |
| 322 | XD-380 | add create() and deploy() methods to TapsController | see https://docs.google.com/a/gopivotal.com/drawings/d/1kCNbVspRBjGc10itF9cSkwST8WgCjgg3wGzFKqLCAzU/edit create TapsController if necessary | 1 |
| 323 | XD-381 | add create() and deploy() methods to TapDeployer | see https://docs.google.com/a/gopivotal.com/drawings/d/1kCNbVspRBjGc10itF9cSkwST8WgCjgg3wGzFKqLCAzU/edit | 1 |
| 324 | XD-382 | Create TapRepository | see StreamsRepository as an example. This includes in memory and Redis implementations | 1 |
| 325 | XD-383 | Implement list() method on TapController | NULL | 1 |
| 326 | XD-384 | Implement list() method on TapDeployer() | NULL | 1 |
| 327 | XD-385 | Error handling on Streams | Have proper exceptions for common error cases on Stream creation/deployment and propagate those to clients correctly. | 5 |
| 328 | XD-386 | Automate copyright header management | Some (java) files are currently missing headers. The plugin at https://github.com/hierynomus/license-gradle-plugin can help, but initial trial revealed that: - skipExistingHeaders does not seem to be honored. We may then need to use a year construction like 2001-${current} or force all files to have ${current} year. Don't know the legal implications of this - Default source sets encompass all files "in the classpath" basically, so that means .xml as well as .properties files for example. It would seem logical to add header to those as well, but I don't think this is what we do on other projetcs. | 3 |
| 329 | XD-388 | Create design document for implementation strategy for ingesting data from twitter into HDFS that can be analyzed by HAWQ | As part of the Hadoop World demonstration work, the flow of data using XD from twitter to be analyzed by HAWQ as done. Part of this work had the data going into HDFS that HAWQ was able to query using external tables. The work for this story is to identify the concrete technical tasks/stories to be created do deliver and document this functionality in XD. | 3 |
| 330 | XD-392 | Create a stubbed out job controller | NULL | 1 |
| 331 | XD-393 | Allow Streamserver/tomcat to chooce free port | Currently you need to define a port which tomcat will use to bind to. If XD is embedded or run in a Hadoop it is not possible to know which port should be used. Current embedded tomcat version is able to use the 0-port trick for it to choose free port. Real binded port can be asked from a connector using below example. org/springframework/xd/dirt/stream/StreamServer.java: public int getLocalPort() { return this.tomcat.getConnector().getLocalPort(); } Also I believe Streamserver should provide a method to ask what is the real url to connect to or atleast what streamserver thinks what it is. This will make life much easier for components empedding spring-xd. | 1 |
| 332 | XD-395 | The shell introduced a fundamental new way to interact with the system, all use of CURL in examples needs to change to use the shell. | NULL | 20 |
| 333 | XD-396 | Add section to documentation that shows command line options available for each server | This should likely be in the "start the runtime" section of Getting Started section. | 1 |
| 334 | XD-397 | Document Monitoring & Management Features | This section should discuss what is exposed via JMX, how you can view it in JConsole, and how you can view it over http via Jolikia. in particular showing how some existing metrics for inbound message channel adapters or the 'inbound' channel of the stream, that indicate the number of messages processed per section. | 2 |
| 335 | XD-398 | Update Getting Started chapter to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#getting-started | 1 |
| 336 | XD-399 | Update Getting Started chapter to include a section on starting the shell. | The chapter on how to start up the shell should ocme right after "start the runtime" and before "create the stream" | 1 |
| 337 | XD-400 | Update Streams Chapter to use shell commands instead of curl | the current streams chapter http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#streams shows creation and deleting streams using CURL - switch to use shell. Also add listing of a stream. there is also an example of creating a stream, this should be replaced as well. | 1 |
| 338 | XD-401 | Create a shell command to post data to an http port for use with the http source module | the current streams chapter http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#streams shows using curl to post some data to a http source module, curl -d "hello" http://localhost:9000 create a shell command so curl doesn't have to be used. https://github.com/SpringSource/rest-shell has a command already developed for this. | 1 |
| 339 | XD-402 | Create a command to browse the HDFS file system | There are existing commands that can be taken from https://github.com/SpringSource/spring-hadoop-samples or https://github.com/SpringSource/impala that can be used for this | 4 |
| 340 | XD-403 | Update Sources section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#http | 1 |
| 341 | XD-404 | Update documentation section "Running in Distributed Mode" to show use of RabbitMQ in addition to Redis | The documentation in the Running in Distributed Mode chapter should discuss that the distributed runtime can use essentially any middleware to communicate between nodes. This functionality is provided by the core ChannelRegistry abstraction. A new intro paragraph shoul convey that it isn't a 'redis' only or 'rabbitmq' only system. There should be "Installing RabbitMQ" and "Starting RabbitMQ" sections to match those for Redis. "Starting Spring XD in Distributed Mode" should cover how to configure the system to select to use Redis or Rabbit. | 3 |
| 342 | XD-405 | Update Sources tail section to use Shell commands instead of curl | http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#tail | 1 |
| 343 | XD-406 | Update Sources twitter search section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#twittersearch | 1 |
| 344 | XD-407 | Update Sources Gemfire CQ section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#gemfire-cq | 1 |
| 345 | XD-408 | Update Source Syslog section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#syslog | 1 |
| 346 | XD-409 | Update Sources TCP section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#tcp | 1 |
| 347 | XD-410 | Update Processors Filter & JSon Filed Value Filter section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#filter http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#json-value-filter | 1 |
| 348 | XD-411 | Update Processors Transform section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#transform | 1 |
| 349 | XD-412 | Update Processors JSON Field Extractor section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#json-field-extractor | 1 |
| 350 | XD-413 | Update Processors Script section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#script | 1 |
| 351 | XD-414 | Update Sink's Log section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#log_sinks | 1 |
| 352 | XD-415 | Update Sink's File section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#file_sinks | 1 |
| 353 | XD-416 | Update Sink's HDFS section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#hdfs | 1 |
| 354 | XD-417 | Update Sink's TCP section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#tcp_sinks | 1 |
| 355 | XD-418 | Update Sink's GemFire section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#gemfire | 1 |
| 356 | XD-419 | Taps introduction section should show use of shell to create a real stream and a real tap using the shell | See http://static.springsource.org/spring-xd/docs/1.0.0.M1/reference/html/#taps The existing docs should be made to show a real stream being created with filter and/or transformer and then a tap that goes to logging. The shell syntax to also stop/undeploy a tap should be shown here as well since the lifecycle is discussed. | 2 |
| 357 | XD-420 | Documentation for AggregateCounter | Add section to Analytics chapter on use of AggregateCounter. The example should show the use of the shell to create the tap that uses the AggregateCounter. | 3 |
| 358 | XD-421 | Update Analytics Counter section to use Shell commands instead of curl | NULL | 1 |
| 359 | XD-422 | Update Analytics Field Value Counter section to use Shell commands instead of curl | NULL | 1 |
| 360 | XD-423 | Update Analytics Gauge section to use Shell commands instead of curl | NULL | 1 |
| 361 | XD-424 | Update Analytics Rich Gauge section to use Shell commands instead of curl | NULL | 1 |
| 362 | XD-425 | Update Samples syslog ingestion section to use Shell commands instead of curl | NULL | 1 |
| 363 | XD-426 | Update Creating a Processor Module section to use Shell commands instead of curl | "test the deployed module" sub-section uses curl. | 1 |
| 364 | XD-427 | Update Creating a Source Module section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.x-SNAPSHOT/reference/html/#_test_the_deployed_module | 1 |
| 365 | XD-428 | Update Creating a Sink Module section to use Shell commands instead of curl | See http://static.springsource.org/spring-xd/docs/1.0.x-SNAPSHOT/reference/html/#_test_the_deployed_module_3 | 1 |
| 366 | XD-429 | Document time source | time source is used in some examples, but it isn't documented explicitly, eg. --interval option in seconds. | 1 |
| 367 | XD-430 | Use a Different Default Jolokia Port for Admin Vs. Container | Avoid the need for {{--jmxPort=xxxx}} when running both a {{Container}} and {{Admin}} on the same server | 1 |
| 368 | XD-431 | Make String conversion optional with local transport | NULL | 5 |
| 369 | XD-433 | Homogenize Container Initialization Failures | If Redis is not running, the container fails to initialize in {{ContainerMain.launch()}} because the connection factory attempts to eagerly connect. If RabbitMQ is not running, the container fails to initialize in {{AbstractContainerLauncher.launch()}}. Make the failure behavior consistent from a user perspective and add a spring-retry {{RetryTemplate}} to retry container startup. | 5 |
| 370 | XD-434 | Consider removing the Topic/Queues when deleting the Stream | As a user, I'd like to have the option to delete the queues/topics so that we can include an _optional_ attribute as part of the stream destroy command to also clean-up the associated queues/topics. *Notes:* * Spring-AMQP {{RabbitAdmin}} now has a {{getQueueProperties()}} method which returns the number of consumers so it may be possible to use it for this purpose. * Consider the possibility of _producers_ and/or _queues_ still containing data * Consider the scenario even after the topics/queues are cleaned-up, what to do with fanout exchange? *Some Further Thoughts* * Consider using the upcoming Spring AMQP REST API {{RabbitManagementTemplate}} if the timing is not right, we could temporarily invoke the rabbit REST API directly. * Should be optional; perhaps via {{stream destroy foo --clean}} * Should this be done by the admin? Or, via a new plugin handling module undeployments - in the rabbit case, undeploying a consumer would check for us being the last consumer and remove the queue/binding/exchange, since we undeploy left->right, everything can be cleaned up on the consumer side. * Third option would be new methods on the bus {{cleanConsumer}} etc invoked by the {{StreamPlugin}} * Down side of doing it on the admin is that he wouldn't necessarily know which rabbit cluster a stream was deployed to - so it probably has to happen on the container - even so, we'd need the admin url(s) for the cluster. | 5 |
| 371 | XD-435 | Create tests to load the standard runtime app context configurations | The basic launch configurations should be tested automatically to ensure that startup scripts and launch aren't broken by changes. | 2 |
| 372 | XD-437 | Fix Package Tangles | Looking at the latest Sonar run we have 3 package tangles in Spring XD: https://sonar.springsource.org/drilldown/measures/7717?metric=87 | 4 |
| 373 | XD-438 | More DSL work: exploiting source/sink channels | The DSL changes under XD-369 now build stream Ast objects that can include a source and sink channel: {code} // Source Channel :mystream.foo > count | log // Sink Channel http | count > :foo {code} These new fields in the Ast object need to be copied into the module deployment request objects and then used at the destination as the channels for wiring things together. Currently the only channels used are the .NNN numeric channels where NNN is the index of the module in the stream definition. The source/sink channels are 'extra' channels that need creating - the source channel acting as a real source for the next module in the chain whilst the sink channel acts as a sink output for the last channel in the chain. I can think of two ways to handle the implementation: - In order to police the stream structure as "source | processor* | sink" maybe special SourceChannel and SinkChannel modules are created to represent these channels and when deployment happens the deployer understands that they don't represent a real request to deploy a module but simply the channels to wire up to the adjacent module. - Carry source/sink channel info in the existing module definitions. But then the verification of source/processor*/sink structure will need modification to say a source isn't necessary if the first processor has a source channel attached and a sink isn't necessary if the last processor has a sink channel attached. | 4 |
| 374 | XD-439 | More DSL work: hooking up stream directory | Following stream parsing there is now a stream resolution stage that chases down substream references and fills in parameterization. The 'lookup' of streams is done through implementors of the StreamLookupEnvironment interface. Currently the parser implements this itself but it is really a job for the stream directory. The parser implementation doesn't know about stream deletions, for example, so may still resolve streams that no longer exist. | 2 |
| 375 | XD-440 | More DSL work: using and policing & for job step lists | The new parser supports | for connecting regular modules and & for connecting job steps. The modules in the ast that were connected with & are tagged but nothing is currently using that information (it doesnt get into the module deployment request). We need to think about using this data: policing the modules that are being deployed to ensure they are job steps, for example. | 3 |
| 376 | XD-441 | More DSL work: checking behaviour for non-deployable streams | With support for substreams/parameterized streams now in the parser it will be possible to create a stream that cannot be deployed: it may not fit the source/processor*/sink structure or it is a parameterized stream with no default values for parameters. Need to check how XD is going to handle these - after creating them, attempting to 'deploy' them should return appropriate errors. (They should exist in the stream directory). | 3 |
| 377 | XD-442 | More DSL work: Documentation updates for new format | Once issues like XD-438 have been completed the wiki doc will need updates to reflect the current behaviour and syntax options. | 2 |
| 378 | XD-443 | Add MQTT Source | NULL | 3 |
| 379 | XD-444 | Make Redis/Rabbit @Rules Conditional | Dependent servers should be required on the CI server, but optional on developer systems. | 1 |
| 380 | XD-445 | Add support to set the read timeout for http request | We need to have the ability to set read timeout for http request. This is already implemented here: https://github.com/SpringSource/rest-shell/ | 1 |
| 381 | XD-446 | Investigate stream lifecycle issues with redis store | When running with "./gradlew launch" I get a 500 error when I try to re-deploy an undeployed stream. A subsequent create or destroy also fail http://localhost:8080:>stream create --definition "time | log" --name ticktock Created new stream 'ticktock' http://localhost:8080:>stream undeploy --name ticktock Un-deployed stream 'ticktock' http://localhost:8080:>stream deploy --name ticktock 13:02:09,936 WARN Spring Shell client.RestTemplate:524 - PUT request for "http://localhost:8080/streams/ticktock" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error http://localhost:8080:>stream deploy --name ticktock 13:03:11,453 WARN Spring Shell client.RestTemplate:524 - PUT request for "http://localhost:8080/streams/ticktock" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error http://localhost:8080:>stream undeploy --name ticktock 13:03:54,576 WARN Spring Shell client.RestTemplate:524 - PUT request for "http://localhost:8080/streams/ticktock" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error http://localhost:8080:>stream undeploy --name ticktock 13:04:48,872 WARN Spring Shell client.RestTemplate:524 - PUT request for "http://localhost:8080/streams/ticktock" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error http://localhost:8080:>stream create --definition "time | log" --name ticktock 13:04:52,066 WARN Spring Shell client.RestTemplate:524 - POST request for "http://localhost:8080/streams" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error http://localhost:8080:>stream destroy --name ticktock 13:05:14,207 WARN Spring Shell client.RestTemplate:524 - DELETE request for "http://localhost:8080/streams/ticktock" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error | 1 |
| 382 | XD-447 | Add an MQTT Sink | NULL | 1 |
| 383 | XD-448 | The user needs the ability to set up a end-time where the trigger should no longer be in effect. | NULL | 1 |
| 384 | XD-449 | The user needs the ability to set up a misfire policy for a Trigger | 2 options are: 1) Fire the trigger immediate - Launch the job when trigger can gather the resources necessary start the job 2) Do nothing - Ignore this job fire time and catch this scenario can occur if XD is down or resources (threads) are not available at the time a job is to be launched. | 1 |
| 385 | XD-450 | Retrieve description of all registered modules by type | The following will retrieve the names of all module types (eg- sources, sinks, jobs, processors, triggers). {code} GET /module-types/ {code} I'm expecting that the plural would be used, but singular would work as well. The following gets modules of a given type: {code} GET /module-types/{type} {code} This would be similar to the {{/modules/}} call in XD-265, but it would only return modules of the specified type. | 5 |
| 386 | XD-451 | Add a Access-Control-Allow-Origin header to responses in order to support cross-origin requests | If we want spring-xd to support interactions through a UI, then it would make sense that we should support the UI coming from a separate origin. This way, the UI could remain a separate project and be served from wherever we want. So, we would need to add the header to *all* outgoing REST responses. We may also need to add a {{Access-Control-Allow-Methods}} header as well. In the short term, the {{Access-Control-Allow-Origin}} header could be hard coded to a specific url (I'm using http://localhost:9889 for now), but in the long term we would need this configurable. | 5 |
| 387 | XD-452 | User wants the ability to persist the final state of a job (success or failure) | NULL | 1 |
| 388 | XD-453 | Use wants the ability to persist Trigger Context | NULL | 1 |
| 389 | XD-454 | User wants the ability to persist the state of a Trigger Instance | NULL | 1 |
| 390 | XD-455 | User wants the ability to limit the total number of jobs a trigger can have running simultaneously | NULL | 1 |
| 391 | XD-456 | User wants the ability to limit the total number of jobs running simultaneously | NULL | 1 |
| 392 | XD-457 | User wants to setup a priority for triggers. | In the case that there are not enough resources to fire a trigger, the highest priority will be fired first. | 1 |
| 393 | XD-458 | User wants the ability to exclude certain days (like holidays) for a trigger to fire. | Commonly called Calendar support | 1 |
| 394 | XD-459 | The user needs the ability to pause and resume triggers ad-hoc. | A pause means that a trigger will wait to fire its job until after the pause is removed. It does not apply the misfire behavior. | 1 |
| 395 | XD-460 | The user needs the ability to pause and resume triggers based on a calendar. | NULL | 1 |
| 396 | XD-461 | User wants to be able to know what triggers are associated with a job | NULL | 1 |
| 397 | XD-462 | User wants a list of currently executing jobs | NULL | 1 |
| 398 | XD-464 | Deploy Spring XD on Hadoop YARN | This is a master ticker tracking work enabling XD to run on Hadoop as Yarn application. | 1 |
| 399 | XD-465 | Shell should display error messages returned from the server | For example, using tcpdump I can see both an exception and message information: 'HTTP/1.1 500 Internal Server Error Server: Apache-Coyote/1.1 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Fri, 12 Jul 2013 13:38:26 GMT Connection: close 275 [{"links":[],"logref":"MessageHandlingException","message":"org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.integration.monitor.IntegrationMBeanExporter#0#92e27d20-4c48-4131-866c-91b6cb642ca2'; nested exception is org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [MessageChannelMonitor: [name=nullChannel, sends=0, receives=0]] with key 'xd.tap1:type=MessageChannel,name=nullChannel,index=1,module=log'; nested exception is javax.management.InstanceAlreadyExistsException: xd.tap1:type=MessageChannel,name=nullChannel,index=1,module=log"}] However the client only shows: http://localhost:8080:>tap create --name "tap1" --definition "tap@test1.file | log" --deploy true 14:38:26,113 WARN Spring Shell client.RestTemplate:524 - POST request for "http://localhost:8080/taps" resulted in 500 (Internal Server Error); invoking error handler Error creating tap 'tap1' The error doesn't seem to be logged in the XD Admin server either, so the information is effectively lost. | 2 |
| 400 | XD-466 | Add JSON conversion to tuple | Support toString() to emit JSON by default. Should be backed by a simple strategy to allow the possibility of other representations. Also provide toTuple(String json). This supports seamless mapping JSON<->Tuple in XD | 3 |
| 401 | XD-467 | JMX shouldn't register taps or streams if the creation fails | There's a lifecycle problem when a tap creation fails (e.g. because the DSL syntax is wrong). Subsequent attempts to create the tap will fail with an error: [{"links":[],"logref":"MessageHandlingException","message":"org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.integration.monitor.IntegrationMBeanExporter#0#92e27d20-4c48-4131-866c-91b6cb642ca2'; nested exception is org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [MessageChannelMonitor: [name=nullChannel, sends=0, receives=0]] with key 'xd.tap1:type=MessageChannel,name=nullChannel,index=1,module=log'; nested exception is javax.management.InstanceAlreadyExistsException: xd.tap1:type=MessageChannel,name=nullChannel,index=1,module=log"}] Disabling JMX solves the issue. reproduce Create a bad stream definition name 'bad' Try to recreate with the same name, but correct stream definitions. The system will report that the stream already exists. | 3 |
| 402 | XD-469 | Upgrade to spring-data-hadoop 1.0.1.RC1 | spring-data-hadoop 1.0.1.RC1 provides flavors for commonly used Hadoop distros/versions and we should make use of that. | 1 |
| 403 | XD-470 | Create JDBC sink | we need a JDBC sink for writing to HAWQ (using int-jdbc:outbound-channel-adapter and postgresql JDBC driver) | 3 |
| 404 | XD-471 | Batching JDBC channel adapter | we need a batching JDBC channel adapter (int-jdbc:outbound-channel-adapter is not batching statements AFAICT) | 8 |
| 405 | XD-472 | Add spring-xd-hadoop distro specific sub-projects | we need to modify build adding two sub-projects for spring-xd-hadoop: one for hadoop 1.1.2 and one for phd1 (Pivotal HD) to pull in transitive dependencies for correct Hadoop distro | 8 |
| 406 | XD-473 | Modify startup script of xdadmin/xdcontainer to allow specifying hadoop distro to use | we need to modify startup script to use hadoop 1.1.2 as default or phd1 when specified with --hadoopDistro=phd1 | 5 |
| 407 | XD-474 | Create JSON to tab-delimited text transformer script | We need a generic script that can do JSON to tab-delimited text transformation for data written to HDFS/HAWQ external tables. Users should be able to specify columns/fields to be included. | 8 |
| 408 | XD-475 | Avro sink for HDFS | We need a sink that can write data in Avro serialized format. This story is for investigating what we would need to do to support that. The Spring Integration Kafka adapter provides Avro support for Kafka. | 8 |
| 409 | XD-476 | The stream definition is not deleted in redis container when the stream is destroyed | This only happened with distributed mode that uses redis as store. The single mode which uses in memory store works fine. Step to reproduce: Create stream: curl -X POST -d "name=httptest" -d "definition=http|log" http://localhost:8080/streams redis 127.0.0.1:6379> keys *httptest* 1) "modules:httptest" 2) "streams.httptest" 3) "stream.definitions.httptest" Delete Stream: curl -X DELETE http://localhost:8080/streams/httptest redis 127.0.0.1:6379> keys *httptest* 1) "streams.httptest" 2) "stream.definitions.httptest" stream still there not deleted Recreate the stream curl -X POST -d "name=httptest" -d "definition=http|log" http://localhost:8080/streams Got: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><errors xmlns:ns2="http://www.w3.org/2005/Atom"><error logref="StreamAlreadyExistsException"><message>There is already a stream with name 'httptest'</message></error> | 2 |
| 410 | XD-477 | Support pagination in list() command for streams | Spring HATEOAS is here to help. Nonetheless, there are currently a number of outstanding issues, namely: https://github.com/SpringSource/spring-hateoas/pull/98 https://jira.springsource.org/browse/SPR-10262#comment-91685 https://github.com/SpringSource/spring-hateoas/pull/94 Creating a issue here for future reference | 5 |
| 411 | XD-478 | Add accepted type logic to module | A module can declare one ore more payload types it will accept. This will inform the runtime re. automatic payload conversion. This can be done in the module XML configuration and processed by StreamPlugin | 3 |
| 412 | XD-479 | Add conversion support to ChannelRegistrar and ChannelRegistry | Implements automatic conversion. Provide APIs to channel registry to register Payload conversion. Includes Redis and Rabbit transport. | 8 |
| 413 | XD-480 | In certain scenarios a job can be redeployed more than once | In a scenario where we are using the same job definition i.e. Job.xml and we create Job Instance Foo. If I create and deploy Foo2 using Job.xml I will see only 2 job definitions(correct), but I will see the job run 3 times. If I create Foo3 & deploy, I will see 3 job definitions(correct), but the jobs will run 5 times. | 2 |
| 414 | XD-481 | Running Job with time delay (non cron) launches 2 instances before job is supposed to fire | NULL | 2 |
| 415 | XD-482 | When creating a job with the fixed-delay parameter in the shell command fails | Need to change the name from fixed-delay to fixed_delay or fixedDelay. System rejects the '-'. | 1 |
| 416 | XD-483 | Job Delete/Destroy Command for shell | NULL | 2 |
| 417 | XD-484 | Create required infrastructure to easily perform integration testing of shell commands | NULL | 10 |
| 418 | XD-485 | Create stories to enable the use of Spring Shell's 2.0 branch testing facilities | We need a few steps 1. Investigate if we need to move off Spring Shell 1.0 dependency, e.g. need to use code in Spring Shell 2.0 branch 2. If we need to use code in Spring Shell 2.0 branch, we need to release a Spring Shell 1.1 M1 release with appropriate code changes. Create stories related to Shell release. 3. Determine and document the basic recipe for doing integration tests. 4. Create stories to provide integration tests for each existing command | 5 |
| 419 | XD-486 | All controllers to return XYZResource objects not the raw domain objects. | Resource objects should be returned from all controller methods. MVC Tests should be added to check returned values. | 2 |
| 420 | XD-487 | StreamsController to return paged results for list() | See implementation used for Steams and apply to jobs, taps, triggers. | 4 |
| 421 | XD-488 | Rename controllers to have pluralized named (e.g. JobsController) | See implementation used for Steams and apply to jobs, taps, triggers. | 1 |
| 422 | XD-489 | Introduce distinction between TapDefinition and Tap (the instance) | Rename existing Tap class to something else. | 3 |
| 423 | XD-490 | StreamDeployer to implement ResourceDeployer | StreamController to not access the repository instance directly, all access to go through StreamDeployer | 4 |
| 424 | XD-491 | Exception Consistency | Favor using custom exceptions instead of using Assert.notNull, review usage and make changes. Eg. if a stream can't be found (or another definition) a 'XYZNotFoundException' instead of Assert.Null on the return value of a findOne method | 3 |
| 425 | XD-492 | Rename create to 'save' in ResourceDeployer | To be consistent with Spring Data Repository method names. | 1 |
| 426 | XD-493 | Remove duplicate code in ResourceDeployer implementations, create abstract base class. | Only ‘create’ in TapDeployer has some additional code to check if the stream exists, could take place in another location. | 4 |
| 427 | XD-494 | Implement common set of controller methods | Save : Save a XYZDefinition - method used to be 'create' Delete : Delete a XYZDefinition - method used to be called 'destroy' Deploy : Deploy a XYZDefinition Undeploy : Undeploy a XYZDefinition List : List a XYZDefinition Returns PagedResources<XYZDefinitionResource> Display : Get specific information about a XYZDefinition Create other stories for each Controller and include in this weeks sprint | 2 |
| 428 | XD-495 | Document MQTT Source and Sink | NULL | 1 |
| 429 | XD-496 | Disable Collection to Object conversion in DefaultTuple | DefaultFormattingConversionService provides Collection -> Object conversion which will produce the first item if the target type matches. Here, this results in an unfortunate side effect, getTuple(List<Tuple> list) would return a Tuple which is misleading. In this case it is preferable to treat it as an error if the argument is not a Tuple. | 3 |
| 430 | XD-497 | Fix Sonar build! | Caused by that weird annotation dependency problem that I worked around for compile. But Sonar complains. One solution would be to add Jackson 2 to the Sonar "classpath", but I did not manage to do that. | 3 |
| 431 | XD-498 | Better UX when admin is not running | Current behavior is to just have a prompt of "unknown:>" I think any return value of a @CliCommand method is not shown b/c the whole infrastructure is not up at that time | 1 |
| 432 | XD-499 | Cron Jobs stop firing when a named trigger is created and deployed | NULL | 3 |
| 433 | XD-500 | If a job is created that uses a trigger that has not been created and deployed it throws a 500 error instead of a 400 | NULL | 2 |
| 434 | XD-501 | Stream Plugin cleanup | public StreamPlugin(){ postProcessContextPath = CHANNEL_REGISTRY; } Subclasses should not directly update superclass fields. | 1 |
| 435 | XD-502 | Delete a trigger from Shell | Shell command to delete a trigger. Note: this command will only remove the trigger definition not modifying the jobs that use the trigger. | 1 |
| 436 | XD-503 | Jobs are created even though they have an invalid definition | Also, if you deploy the Job it will fail, but then you can't delete the job. | 2 |
| 437 | XD-504 | Add "How to Build Spring-XD" instructions to the documentation | We need to determine where this information could fit in. It can be either in "README" at the project home page or "Getting started" wiki page. | 1 |
| 438 | XD-505 | Fix XD modules parameters with "-" to use camel case | XD-482 addresses the use of camel case in 'fixed-delay' job module parameter name. and, we need to fix the same for other module parameters wherever '-' is being used. | 2 |
| 439 | XD-506 | Support pagination in list() command for jobs | See XD-477 | 1 |
| 440 | XD-507 | Support pagination in list() command for taps | See XD-477 | 1 |
| 441 | XD-508 | Support pagination in list() command for triggers | See XD-477 | 1 |
| 442 | XD-509 | Investigate intermittent failure of RedisStreamDefinitionRepositoryTests | the test for findAll often fails for me when running inside gradle. (Could not reproduce inside eclipse) I already tried fixing it by using a different redis key space, but to no avail. One explanation would be if gradle runs tests concurrently, but my understanding is that it does not. | 3 |
| 443 | XD-510 | Ensure that each controller's list() returns PagedResources | NULL | 4 |
| 444 | XD-511 | Create XD integration test framework | Add top level utility methods to manage XD runtime (deploy, start and stop). These methods will be used by underlying integration tests to control runtime test environment. | 2 |
| 445 | XD-512 | Create XD shell integration test parser | Since most of the xd.shell.itests will do more then one thing (deploy a stream, start it, add a tap, add a job, stop, etc) we decided to decouple writing testcases with running it. Test cases will be written in spring-shell scriptlets. Scriptlets are json files with command and expectedResult as tokens. Here's an example: {code} { "testscript": [ {"command" : "stream create --definition "http | file" --name http2file", "result" : "Created new stream 'http2file'"}, {"command" : "stream list", "result" : "..."} ] } {code} A parser will parse scriptlets, executes it and asserts on expected results. | 3 |
| 446 | XD-513 | Add CI job in bamboo to run XD integration tests | CI job will run integration tests that are tagged for CI build. | 1 |
| 447 | XD-514 | Create proper test coverage for Controllers | Create proper test coverage for Controllers | 8 |
| 448 | XD-515 | Address already in use for tomcat/hsqldb should fail completly | Currently, process is left running and if logs / sysout are not monitored, you have no clue | 3 |
| 449 | XD-516 | Admin should fail immediately if Rabbit is not running | Container currently does that. | 3 |
| 450 | XD-517 | Add Server Runtime Info to Banner | Standalone Admin currently has no shiny banner as container has. More importantly, it does not say which port it's listening on, the transport used, etc. | 3 |
| 451 | XD-518 | Upgrade to Jackson 2.2.2 | NULL | 2 |
| 452 | XD-519 | Modules need to validate their parameters at create time. | We need to fail fast. | 3 |
| 453 | XD-520 | All parameters for modules need to use "hump case" formerly camel hump | NULL | 1 |
| 454 | XD-521 | The parser should be able to handle a parameter name with a '-' hyphen embedded. | Right now it treats it as a new parameter start and fails. | 1 |
| 455 | XD-522 | Cannot create tap if you have already tried to create an invalid one of same name | From the shell: {code} > stream create --name aaa --definition "time|log" Created new stream 'aaa' > tap create --name aa --definition "tap aaa | log" Command failed org.springframework.xd.rest.client.impl.SpringXDException: XD111E:(pos 8): Unexpected token. Expected 'dot(.)' but was 'pipe(|)' tap aaa | log >tap create --name aa --definition "tap aaa . log" Command failed org.springframework.xd.rest.client.impl.SpringXDException: There is already a tap named 'aa' {code} Looks like the first tap was created even though there was a parse error. And so the second attempt to create the tap failed due to an existing tap. | 1 |
| 456 | XD-523 | Parser blows on modules names with '-' | Tried to create a module named "tcp-poll" and got this: XD108E:(pos 3): missing expected character '-' tcp-poll --host=54.208.22.193 --port=8081 | log I believe this should be supported, and indeed we have several module names of this form already | 5 |
| 457 | XD-524 | Update jobs section to use shell | NULL | 1 |
| 458 | XD-525 | Provide .settings formatting rules so that they're shared | Thinking about using the official SpringSource rules as a template | 3 |
| 459 | XD-526 | Make module files classpath aware | Currently, living at the root of the project, those files don't benefit from IDE SI awareness. Make it so that they belong to a java project which sees the correct version of the SI jars used. Has impact on the build.gradle file | 5 |
| 460 | XD-527 | Create new implementations of existing infrastructure (syslog adapters and TaskExecutors) | NULL | 10 |
| 461 | XD-528 | Create SI components that wrap Reactor's TCP server | NULL | 4 |
| 462 | XD-529 | Review existing Reactor syslog codec implementation | NULL | 2 |
| 463 | XD-530 | Create XD module for syslog-tcp-reactor | Still keep existing one. | 3 |
| 464 | XD-531 | Check for high CPU usage with syslog-tcp-reactor module | NULL | 2 |
| 465 | XD-532 | Adapt SpringOne 2012 UI code from keynote demo of election results to use XD | Existing code: https://github.com/ghillert/springone2012 | 6 |
| 466 | XD-533 | Create shell integration tests for stream lifeycle | create, delete, deploy streams... | 2 |
| 467 | XD-534 | Create shell integration tests for tap lifeycle | creating taps, deleting | 2 |
| 468 | XD-535 | Create shell integration tests for trigger lifeycle | creating triggers, deleting triggers | 4 |
| 469 | XD-536 | Create shell integration tests for job lifeycle | creating job defs, deploying jobs, undeploying jobs, deleting job defs | 4 |
| 470 | XD-538 | Develop infrastructure to enable testability of commands | This requires to boostrap the singlenode admin server in process, submit commands to the shell programmatically, and assert on the results of executing the command. | 5 |
| 471 | XD-539 | Investigate using Redis txs and pipeline for Inbound/Outbound Q Adapter perf improvements | Investigate using transactions and pipelining to improve performance of both the inbound and outbound RedisQueue channel adapters. Involves testing against a pre-release of SDR 1.1 M2. | 5 |
| 472 | XD-540 | Broadcast Undeploy Requests | Use an 'undeploy' topic to broadcast undeploy requests to all containers. Applies to Redis and Rabbit transports, not local. Also, rename {{ModuleDeploymentRequest}} to {{ModuleOperationRequest}} with an enum {{DEPLOY}}, {{UNDEPLOY}}. | 5 |
| 473 | XD-541 | Add Message Source for error messages returned to users | NULL | 3 |
| 474 | XD-542 | Refactor Module to Encapsulate Group and Index | Currently many methods take module, group, index - defining a module instance; group and index can be encapsulated in {{Module}} so one arg can be passed around. | 2 |
| 475 | XD-543 | Handle Pagination in Spring XD Shell | NULL | 5 |
| 476 | XD-544 | Fix In-Memory Analytics | Most of the infrastructure and code cleanup has been done for In-Memory Analytics. The only remaining issue is that, by including memory-analytics.xml from common.xml, we're actually creating e.g. a new InMemoryCounterRepository that is different from the one present in the Admin process space. This story involves fixing that. It may actually be done as part of XD-353, handling the "local" transport as a special case (context inheritance) rather than import based on xd.transport | 3 |
| 477 | XD-545 | Display a counter | NULL | 2 |
| 478 | XD-546 | Display a Field Value Counter | NULL | 2 |
| 479 | XD-547 | Display an Aggregate Counter | NULL | 3 |
| 480 | XD-548 | Display a Gauge | NULL | 1 |
| 481 | XD-549 | Display a Rich Gauge | NULL | 2 |
| 482 | XD-550 | Leverage fieldNameToCounterNameMap in FieldValueCounterHandler | FieldValueCounterHandler was first written to support setting several fields at once, but the current constructor / field-value-counter.xml does not use it. Either leverage that feature or simplify code | 3 |
| 483 | XD-551 | Add "trigger list" support to Spring XD Shell | NULL | 1 |
| 484 | XD-552 | Add status column for 'stream list' shell command result | {{stream list}} shell command should display status of the stream (deployed, undeployed) | 1 |
| 485 | XD-553 | Add additional options to File source | Seems like the current file source results from an initial POC. Very few things can be parameterized, including the polled directory that needs to be in /tmp/xxx To be useful in production, we might want to revisit | 3 |
| 486 | XD-554 | Separate Module Context Refresh from Context Start | Split plugin module processing into pre and post processing where preprocessing is done before the context is refreshed and post processing is done after the refresh, but before the start. In the Stream plugin, wire the module into the {{ChannelRegistry}} during post processing, instead of using the {{ChannelRegistrar}}. | 5 |
| 487 | XD-555 | make application/json the default output type for the REST API? | the top-level URL works via simple curl (without an Accept header) or the browser: {code} > curl http://localhost:8080 {"links":[{"rel":"streams","href":"http://localhost:8080/streams"},{"rel":"triggers","href":"http://localhost:8080/triggers"},{"rel":"jobs","href":"http://localhost:8080/jobs"},{"rel":"taps","href":"http://localhost:8080/taps"},{"rel":"counters","href":"http://localhost:8080/metrics/counters"}]} {code} However, trying any of those links then fails, e.g.: {code} > curl http://localhost:8080/streams <?xml version="1.0" encoding="UTF-8" standalone="yes"?><errors xmlns:ns2="http://www.w3.org/2005/Atom"><error logref="HttpMessageNotWritableException"><message>Could not marshal [PagedResource { content: [links: [<http://localhost:8080/streams/mqttdemo>;rel="self"]], metadata: Metadata { number: 0, total pages: 1, total elements: 1, size: 20 }, links: [] }]: null; nested exception is javax.xml.bind.MarshalException - with linked exception: [com.sun.istack.internal.SAXException2: unable to marshal type "org.springframework.xd.rest.client.domain.StreamDefinitionResource" as an element because it is not known to this context.]</message></error></errors> {code} | 2 |
| 488 | XD-556 | CORS support | XD instances will not accept xhr requests from browsers whose page origin does not match the XD instance. As an example, the Kodiak UI is served from a different process (and url) than the XD instance. When users open the Kodiak UI in a browser, requests from the browser to the XD instance, but these requests will fail due to cross-site scripting limitations. CORS (Cross-Origin Resource Sharing) is a way to get around this. We can configure the server to accept requests from browsers whose origins are not the XD instance. I have this working in a local branch and will submit a pull request. More information: CORS Spec: http://www.w3.org/TR/cors/ SPR-9278 CORS support for SpringFramework | 5 |
| 489 | XD-557 | HTML Doco has font issues for [source] | It seems the html rendering of documentation is using a variable width font for some of the code (esp. [source,sh] apparently) rendering. Weird thing though is that when mouse hovering over some of them, they went back to fixed width. See screenshot. http://static.springsource.org/spring-xd/docs/current-SNAPSHOT/reference/html/#_start_the_runtime_and_the_xd_shell | 3 |
| 490 | XD-558 | Create IntegralMetric and IntegralResource types | There is some overlap between the gauge and counter repository types and also between the domain resources used by the REST controllers (CounterResource, GaugeResource). The aggregate counter may wish to return a simple count value for the counter too, which would also be a simple integral value. | 3 |
| 491 | XD-559 | Send failing sonar build message to spring-xd mailing list. | NULL | 1 |
| 492 | XD-560 | Ad-hoc Jobs do not start | After XD-554 the ModuleJobExecutor.start does not get called. Thus jobs that are "run one time" do not fire. | 3 |
| 493 | XD-561 | Failure when creating/deploying stream leaves invalid stream registry/definitions in the Repository implementations. | reproduce 1) Create a bad stream definition name 'bad' Try to recreate with the same name, but correct stream definitions. The system will report that the stream already exists. | 4 |
| 494 | XD-562 | Documentation for use of conversion service and creating custom processing modules that use the Tuple data structure. | NULL | 2 |
| 495 | XD-563 | Regression test existing functionality of stream/taps based on introduction of new conversion functionality | make sure nothing is broken - spot check using. 1) ticktock 2) twitter 3) gemfire | 2 |
| 496 | XD-564 | Upgrade sink and processor modules to use new conversion service | NULL | 2 |
| 497 | XD-565 | Documentation for using a specific Hadoop distribution | Show how to select a specific hadoop distribution when starting embedded/standalone XDContainer. | 1 |
| 498 | XD-566 | Document JDBC module | NULL | 1 |
| 499 | XD-567 | Documentation for fixed rate triggers | NULL | 1 |
| 500 | XD-568 | Documentation for deleting triggers | NULL | 1 |
| 501 | XD-569 | Investigate failures to start a stream when using named channels. | Create a reproducible series of steps or shell integration test. | 2 |
| 502 | XD-571 | Create shell integration test for named chanels | Expected usage (ATM) would be // sink channel called foo http | transform --expression=payload.toUppercase() > :foo // source channel called foo :foo > count | file | 3 |
| 503 | XD-572 | Prepare Blog post for XD M2 | NULL | 5 |
| 504 | XD-574 | Replace usage of 'raw' curl with shell command to post http data in documentation | e.g. http://localhost:8080:>post httpsource --target http://localhost:9090 --data 10 I believe this will also help to avoid ugly syntax to escaping quotes for json as in the gemfire example. | 1 |
| 505 | XD-575 | Change http command to post data by putting 'http' as the main command option | The current http command is of the form http://localhost:8080:>post httpsource --target http://localhost:9090 --data 10 It isn't intuitive to think 'post', rather the command can be http post --target http://localhost:9090 --data 10 which will allow us to have support for other http verbs and cleanly separate the namespace from 'hadoop' etc. The RestShell from which this came was only concerned with http actions, so the leading command classification probably seemed superfluous. | 1 |
| 506 | XD-576 | Change banner of shell to say only 'xd' | NULL | 1 |
| 507 | XD-577 | In documentation, replace usage of 'raw' hadoop command with shell 'hadoop' commands | NULL | 1 |
| 508 | XD-578 | Can't access HDFS using webhdfs protocol | http://localhost:8080:>hadoop config fs --namenode webhdfs://localhost:50070 http://localhost:8080:>hadoop fs ls / Hadoop configuration changed, re-initializing shell... run HDFS shell failed. Message is: org/mortbay/util/ajax/JSON This was on a hadoop 1.0.1 install The hdfs http interface was available $ curl -i "http://localhost:50070/webhdfs/v1/tmp?op=GETFILESTATUS" HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked Server: Jetty(6.1.26) {"FileStatus":{"accessTime":0,"blockSize":0,"group":"supergroup","length":0,"modificationTime":1365015846724,"owner":"mpollack","pathSuffix":"","permission":"777","replication":0,"type":"DIRECTORY"}} | 3 |
| 509 | XD-579 | Modify startup script of xd shell to allow specifying hadoop distro to use | NULL | 1 |
| 510 | XD-580 | XD Shell needs to support multiple Hadoop distros | From https://github.com/SpringSource/spring-xd/pull/161: "The command shell needs to also support different hadoop distribution options. Perhaps the shell just uses a relative path to the location of xd/lib/" | 3 |
| 511 | XD-581 | configuration conflict when using "--transport", "local", "--store", "redis", "--disableJmx", "true", "--analytics", "redis" | results in both in-memory and redis based definitions of RichGaugeService - can't satisfy autowiring because there are two candidates. Had to change --analytics=memory to get the application context to load. | 2 |
| 512 | XD-582 | Support named channels when using local transport | Sending data to an incomplete stream which is created using a named sink channel only works when using Redis (or Rabbit?, not tested). Since the in-memory version doesn't use a queue, it will fail if you are using xd-singlenode. We should use a queue channel with unlimited capacity to allow messages to be sent before the full stream is created. | 4 |
| 513 | XD-583 | Dispatcher Has No Subscriber Error when posting a message to a stream | This has been observed intermittently with Redis transport by myself and others when sending a message to a valid stream. Not sure how to recreate it yet. 11:27:10,082 ERROR ThreadPoolTaskScheduler-1 redis.RedisQueueInboundChannelAdapter:126 - Error sending message org.springframework.integration.MessageDeliveryException: Dispatcher has no subscribers for channel 'org.springframework.context.support.GenericApplicationContext@3f73865d.input'. | 5 |
| 514 | XD-585 | Deploying with twittersearch source throws Jackson ClassDefNotFound exception | The upgrade to Jackson 2.2 included the following change to the build script {code} project('spring-xd-dirt') { description = 'Spring XD DIRT' configurations { [runtime,testRuntime]*.exclude group: 'org.codehaus.jackson' } {code} Spring social twitter template depends on these classes | 1 |
| 515 | XD-586 | Document queue channel capacity configurable when using local transport | NULL | 3 |
| 516 | XD-587 | Create a abstract base class for rest controllers | NULL | 3 |
| 517 | XD-588 | RedisAggregateCounterRepository doesn't give proper results back | Both Luke's original code and my refactored PR[1] (which uses same code snippet) seem to behave strangely. Stored values seem fine, but the getCounts() method seems phony. To test: 1) stream create foo --definition "time|log" 2) tap create bar --definition "tap@foo | aggregatecounter" 3) curl -H "application/json" http://localhost:8080/metrics/aggregate-counters/bar this gives default bucketing (hourly) but chances are that they are empty. | 3 |
| 518 | XD-589 | Create AbstractStreamIntegrationTest that will destory streams that were created during test method execution | Keep track of named streams that were create and use @After to destroy them. | 1 |
| 519 | XD-591 | Improve build file distribution tasks | The current flow of gradle tasks is confusing. Suggest the following changes to simplify the flow. 1. Move the current task logic in zipXD to distZip 2. Have distZip depend on dist 3. Update the 'how to build docs' on the wiki 4. Make sure that the distZip task only shows up once in the list of gradle target. | 1 |
| 520 | XD-592 | Problems with advanced tapping | Start of a test program that can be placed in StreamCommandTests: {code} @Test public void testTappingAndChannels() { executeStreamCreate("myhttp","http --port=9314 | transform --expression=payload.toUpperCase() | log",true); executeStreamCreate("tap","tap @myhttp.1 | log",true); executeStreamCreate("tap_new","tap myhttp.1 > log",true); executeCommand("http post --data Dracarys! --target http://localhost:9314"); // TODO verify both logs output DRACARYS! } {code} In the test program see two taps. One using the older style and one using the newer style and '>' so that there is no real tap module source, the log module just gets its input channel wired directly to myhttp.1 (the output of transform). They should be doing the same thing. However when run the output for tap_new is missing, all I see is: {code} 11:39:36,055 WARN New I/O worker #28 logger.tap:141 - DRACARYS! 11:39:36,059 WARN New I/O worker #28 logger.myhttp:141 - DRACARYS! {code} No errors are reported, there is just no output for tap_new. | 8 |
| 521 | XD-593 | Add "counter delete" shell command | Add "counter delete" shell command. This also requires implementation of DELETE rest end point at CountersController. | 1 |
| 522 | XD-594 | Create list/delete commands for all the metrics | We need to add list/delete commands for the metrics: InMemoryAggregateCounter FieldValueCounter Gauge RichGauge Currently, the AbstractMetricsController class has the delete method to delete the metric from the repository. We can probably use the same for all the metrics. | 2 |
| 523 | XD-595 | Fix wiki documentation to use xd shell command prompt to read "xd:>" | We need to fix the github wiki to use the xd shell command prompt "xd:>". | 1 |
| 524 | XD-596 | Add CONTRIBUTING.md file | Add CONTRIBUTING.md file, use the Spring Integration file as the basis. | 1 |
| 525 | XD-597 | Ad-Hoc Job needs to have option for launch and forget | When running an ad-hoc job without the use of a trigger (adhoc or named). The user has to wait for job to complete before receiving a success. We need to launch a job and get a success back to the user letting them know the job has been launched. for example --immediate | 4 |
| 526 | XD-598 | Gemfire cache closed when a gemfire module is undeployed | Need to investigate why this is happening, normally setting {code:xml} <gfe:client-cache close="false"/> {code} prevents the (singleton) cache from closing when the application context is closed. | 4 |
| 527 | XD-599 | Gradle Import Broken by Hadoop Pseudo Projects | When importing Spring-XD as a gradle project, in STS, while building the model, we get Root exception: java.lang.IllegalArgumentException: Project location doesn't exist: .../spring-xd/spring-xd-hadoop/hadoop11 ./gradlew eclipse creates these directories, but the plugin needs them before running that task The problem seems to be that these "projects" are not really projects. Perhaps a quick fix would be to commit these directories (with a dummy file) ?? | 1 |
| 528 | XD-600 | Add deploy/undeploy/destroy 'all' commands for all applicable resources (streams, tap, job & trigger,) | NULL | 5 |
| 529 | XD-601 | ./xd-container --transport local throws NumberFormatException | ./xd-container [OK] ./xd-container --transport redis [OK] ./xd-container --transport rabbit [OK] ./xd-container --transport local [FAIL] wkoh-mbp:bin administrator$ ./xd-container --transport local Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.jolokia.jvmagent.spring.SpringJolokiaAgent#0': Invocation of init method failed; nested exception is java.lang.NumberFormatException: For input string: "${xd.jmx.port}" org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1488) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461) org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292) org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626) org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) org.springframework.xd.dirt.server.ContainerMain.launch(ContainerMain.java:89) org.springframework.xd.dirt.server.ContainerMain.main(ContainerMain.java:72) Caused by: java.lang.NumberFormatException: For input string: "${xd.jmx.port}" java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) java.lang.Integer.parseInt(Integer.java:449) java.lang.Integer.parseInt(Integer.java:499) org.jolokia.jvmagent.JolokiaServerConfig.initConfigAndValidate(JolokiaServerConfig.java:211) org.jolokia.jvmagent.JolokiaServerConfig.init(JolokiaServerConfig.java:84) org.jolokia.jvmagent.JolokiaServerConfig.<init>(JolokiaServerConfig.java:68) org.jolokia.jvmagent.spring.SpringJolokiaAgent.afterPropertiesSet(SpringJolokiaAgent.java:78) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1547) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485) ... 11 more | 2 |
| 530 | XD-602 | Fix ChannelRegistry Cleanup During Module Undeploy | NULL | 2 |
| 531 | XD-603 | NPE on stream destroy | xd:>stream create ticktock --definition "time | log" --deploy true 18:45:13,310 WARN Spring Shell client.RestTemplate:524 - POST request for "http://localhost:8080/streams" resulted in 400 (Bad Request); invoking error handler Command failed org.springframework.xd.rest.client.impl.SpringXDException: There is already a stream named 'ticktock' xd:>stream destroy ticktock 18:45:16,505 WARN Spring Shell client.RestTemplate:524 - DELETE request for "http://localhost:8080/streams/ticktock" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.xd.rest.client.impl.SpringXDException: java.lang.NullPointerException Caused by: java.lang.NullPointerException org.springframework.xd.dirt.module.ModuleDeployer.undeploy(ModuleDeployer.java:143) org.springframework.xd.dirt.module.ModuleDeployer.handleMessageInternal(ModuleDeployer.java:97) | 2 |
| 532 | XD-605 | Rich Gauge doco is outdated | The RichGauge section does not mention the "alpha" parameter in redis output, nor does it explain its meaning. | 2 |
| 533 | XD-606 | Document JSON quoting behavior in shell | This may be an issue following the search/replace from curl to Shell, but for example, this documentation line does not work: http post --target http://localhost:9000 --data "{\"symbol\":\"VMW\",\"price\":72.04}" The backslash prior to quote is left in the payload (and hence Jackson chokes on it) We need clear rules about quoting at the shell level | 3 |
| 534 | XD-607 | Integration tests for "DSL Reference" examples | NULL | 5 |
| 535 | XD-608 | Fix text-table rendering | NULL | 1 |
| 536 | XD-609 | Documentation for rabbit source | http://static.springsource.org/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#sources should have 'rabbit' added to the list and also the corresponding section that shows some basic usage. | 1 |
| 537 | XD-610 | Documentation for jms source | http://static.springsource.org/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#sources should have 'jms' added to the list and also the corresponding section that shows some basic usage. | 1 |
| 538 | XD-611 | Documentation for file source | http://static.springsource.org/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#sources should have 'file' added to the list and also the corresponding section that shows some basic usage. | 1 |
| 539 | XD-612 | Create a rabbit sink module and documentation | https://github.com/springsource/spring-xd/wiki/Sinks should have 'rabbit' added to the list and also the corresponding section that shows some basic usage. | 2 |
| 540 | XD-613 | Deployed streams should be restarted on container start | When using Redis store, stored deployed streams should be deployed on container restart. | 5 |
| 541 | XD-614 | Conversion Enhancements | Content-Type during transport transit is not the same as the content-type within modules. "Real" transports always use byte[] which may contain raw byte[] from a source, a byte[] converted from a String (which may or may not already contain JSON), or a byte[] containing JSON converted by the transport on the outbound side. The transport needs to convey which of these was applied on the outbound side so it can properly reconstruct the message. Retain any content-type header that already exists in the message, and restore it. For Rabbit, use normal SI/Rabbit headers to convey this information. For Redis, add the information to the byte[]. | 8 |
| 542 | XD-615 | CommandResult return sucess even if hadoop shell command fails | when automating tests for creating http|hdfs stream, I run into an issue where CommandResult object always set success=true even if the actual hadoop shell command fail. == valid hdfs url getShell().executeCommand("hadoop config fs --namenode hdfs://localhost:8020"); CommandResult cr = getShell().executeCommand("hadoop fs ls /"); == output Found 2 items drwxr-xr-x - administrator supergroup 0 2013-08-05 17:18 /user drwxr-xr-x - administrator supergroup 0 2013-08-05 17:18 /xd CommandResult [success=true, result=null, exception=null] == invalid hdfs url getShell().executeCommand("hadoop config fs --namenode hdfs://localhost:8021"); CommandResult cr = getShell().executeCommand("hadoop fs ls /"); == output Bad connection to FS. command aborted. exception: Call to localhost/127.0.0.1:8021 failed on connection exception: java.net.ConnectException: Connection refused CommandResult [success=true, result=null, exception=null] Ideally, we should set success=false if hadoop command fail and if hadoop command succeeds, we should set success=true and populate result= output from hadoop command instead of result=null | 3 |
| 543 | XD-617 | Making an http post with json double quoted will hang the shell. | Was following the (now updated) directions for gemfire-cq source. xd:> stream create --name stocks --definition "http --port=9090 | gemfire-json-server --regionName=Stocks --keyExpression=payload.getField('symbol')" xd:> stream create --name cqtest --definition "gemfire-cq --query='Select * from /Stocks where symbol=''VMW''' | file" xd:> http post --target http://localhost:9090 --data "{"symbol":"VMW", "price":73}" The double quotes were causing a problem with xd-singlenode Aug 06, 2013 5:38:15 PM org.jboss.netty.channel.SimpleChannelUpstreamHandler WARNING: EXCEPTION, please implement org.springframework.integration.x.http.NettyHttpInboundChannelAdapter$Handler.exceptionCaught() for proper handling. org.springframework.integration.transformer.MessageTransformationException: org.springframework.integration.MessageHandlingException: org.springframework.integration.transformer.MessageTransformationException: Expected a ':' after a key at 22 [character 23 line 1] org.springframework.integration.transformer.MessageTransformingHandler.handleRequestMessage(MessageTransformingHandler.java:73) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:137) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) Using single quotes inside the json brackets worked, need to investigate. | 2 |
| 544 | XD-618 | Update to Spring Shell 1.1.0.M1 release | NULL | 1 |
| 545 | XD-619 | Add list command for AggregateCounter | NULL | 2 |
| 546 | XD-620 | Refactor test cases to move away from inheritance model of utility methods for streams, counters | Model the API more akin to SpringXDOperations api. | 2 |
| 547 | XD-621 | Set Default Hadoop Name Node for Shell | Currently, you have to set the default name node every time your start the shell. We should do 2 things: - Provide a default Name node Set Default Hadoop Name Node for Shell: hdfs://localhost:8020 - Should we provide some form of persistence? It kind of sucks that you have to re-specify the name node every time the shell starts up {code} xd:>hadoop fs ls / You must set fs URL before run fs commands {code} | 2 |
| 548 | XD-622 | Fix JavaDoc Warnings | /home/gpr/Documents/github/spring-xd/spring-xd-analytics/src/main/java/org/springframework/xd/store/AbstractRedisRepository.java:196: warning - @param argument "the" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-module/src/main/java/org/springframework/xd/module/AbstractPlugin.java:93: warning - @return tag has no arguments. /home/gpr/Documents/github/spring-xd/spring-xd-rest-domain/src/main/java/org/springframework/xd/rest/client/util/RestTemplateMessageConverterUtil.java:63: warning - Tag @link: reference not found: StreamDefinitionResource.Page /home/gpr/Documents/github/spring-xd/spring-xd-rest-client/src/main/java/org/springframework/xd/rest/client/TapOperations.java:40: warning - @param argument "control" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/server/options/OptionUtils.java:29: warning - @parame is an unknown tag. /home/gpr/Documents/github/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/stream/MissingRequiredDefinitionException.java:38: warning - @param argument "name" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/stream/StreamServer.java:102: warning - @return tag has no arguments. /home/gpr/Documents/github/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/stream/TapDefinition.java:57: warning - @return tag has no arguments. /home/gpr/Documents/github/spring-xd/spring-xd-module/src/main/java/org/springframework/xd/module/AbstractPlugin.java:93: warning - @param argument "group" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-module/src/main/java/org/springframework/xd/module/AbstractPlugin.java:93: warning - @param argument "index" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-module/src/main/java/org/springframework/xd/module/AbstractPlugin.java:101: warning - @param argument "group" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-module/src/main/java/org/springframework/xd/module/AbstractPlugin.java:101: warning - @param argument "index" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/json/TypedJsonMapper.java:133: warning - @return tag has no arguments. /home/gpr/Documents/github/spring-xd/spring-xd-shell/src/main/java/org/springframework/xd/shell/util/UiUtils.java:60: warning - @CloudApplication is an unknown tag. /home/gpr/Documents/github/spring-xd/spring-xd-tuple/src/main/java/org/springframework/xd/tuple/TupleJsonMarshaller.java:26: warning - @param argument "tupleToStringConverter" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-tuple/src/main/java/org/springframework/xd/tuple/TupleJsonMarshaller.java:26: warning - @param argument "stringToTupleConverter" is not a parameter name. /home/gpr/Documents/github/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/stream/dsl/DSLException.java:89: warning - @return tag has no arguments. /home/gpr/Documents/github/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/stream/dsl/StreamLookupEnvironment.java:60: warning - @return tag has no arguments. | 1 |
| 549 | XD-623 | catch erroneous hadoop config fs --namenode url early | currently, we can specify any bogus url using 'hadoop config fs --namenode' without any warning. e.g. hadoop config fs --namenode hdfs://localhost:8888 doing a 'hadoop fs ls /' will catch the error and throw exception. Ideally, we should catch the bogus url error early in the 'hadoop config fs' command. similar to 'admin config server --uri' | 2 |
| 550 | XD-624 | Use External Connection Factory in TCP Syslog Source | WARN log emitted because the embedded connection factory does not get an application event publisher. Will be fixed in SI M3 (INT-3107). | 1 |
| 551 | XD-625 | Revert XD-624 When SI 3.0.M3 is Available | NULL | 1 |
| 552 | XD-626 | Shell: RestTemplate not posting using UTF-8 | The *http post* command uses the default MediaType by the RestTemplate, which in return triggers the default *StringHttpMessageConverter* which itself uses the default charset *ISO-8859-1*. This creates issues when posting special characters. | 1 |
| 553 | XD-627 | shell command "stream list" fails | Run the shell command {{stream list}} and you get the following error: {code} xd:>stream list Command failed org.springframework.http.converter.HttpMessageNotReadableException: Could not read JSON: Unrecognized field "metadata" (class org.springframework.xd.rest.client.domain.StreamDefinitionResource$Page), not marked as ignorable (3 known properties: , "links", "content", "page"]) at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@30721965; line: 1, column: 148] (through reference chain: org.springframework.xd.rest.client.domain.Page["metadata"]); nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "metadata" (class org.springframework.xd.rest.client.domain.StreamDefinitionResource$Page), not marked as ignorable (3 known properties: , "links", "content", "page"]) at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@30721965; line: 1, column: 148] (through reference chain: org.springframework.xd.rest.client.domain.Page["metadata"]) {code} You get a similar error when running any of the following: {code} tap list trigger list job list {code} | 1 |
| 554 | XD-628 | Streams created without a '|' (substreams) are being typed by the parser as a Job | NULL | 3 |
| 555 | XD-629 | Problem with tapping on a module using a named sink channel | {code} mystream = http | transform --payload=expression.toUpperCase() > :foo tap mystream.transform | log {code} This appears to fail because we can't tap into whatever was created to represent the named channel 'foo'. There is an @Ignore test in StreamCommandTests called testTappingModulesVariationsWithSinkChannel() which checks this. (The parser is currently resolving 'tap mystream.transform' to 'tap --channel=foo'.) | 2 |
| 556 | XD-630 | StreamCommandTests - asserting sink contents sometimes failing | There are some asserts in StreamCommandTests that are commented out (see the TODOs in there). These asserts are verifying the contents of the various sinks employed in the tests. I am finding that if the tests are run all together with the asserts enabled (run all StreamCommandTests), some of the assertions fail, with something like: {code} org.junit.ComparisonFailure: expected:<[DRACARYS! ]> but was:<[]> org.junit.Assert.assertEquals(Assert.java:115) org.junit.Assert.assertEquals(Assert.java:144) {code} When run individually the tests succeed. Not sure if it is timing (checking sinks before they've been written to) or something else... | 2 |
| 557 | XD-631 | Pluralize test classes in package org.springframework.xd.shell.command | The classes under test are pluralized. Therefore, the test classes themselves should reflect that. E.g. rename *JobCommandTests* to *JobCommandsTests* as it tests class *JobCommands*. Please check all tests in that package for correct naming. | 1 |
| 558 | XD-632 | Shell: HTTP Post - Allow posting of local file contents | E.g. allow for posting of JSON data stored in local files. * Allow users to specify the *content-type*. * Ensure that Unicode data (UTF) posts correctly. | 4 |
| 559 | XD-633 | Rabbit Source - Make Default QueueName == Stream Name | Consistency with JMS Source. | 1 |
| 560 | XD-634 | Fix guava dependency for hadoop20 and phd1 | Spring Xd currently ships with Guava 12.0 while Hadoop 2.0.5 and Pivotal HD 1.0 depends on 11.0.2 - this could lead to classpath problems if we include both. | 3 |
| 561 | XD-635 | OOTB source modules with poller should use fixed-delay | pollers should standardize on fixed-delay vs fixed-rate. The value should accept a property with a standard name like 'interval' | 1 |
| 562 | XD-636 | x-xd-* Transport Content-Type Leakage | The {{AbstractReplyProducingMessageHandler}} in the Rabbit transport exposes the internal transport content-type, if none existed on the original transported message. | 1 |
| 563 | XD-637 | Channel Registry Refactoring | Factor out common Redis/Rabbit {{ChannelRegistry}} code. Also, factor out common inbound/tap code (very similar). Change transport nternals to Use AbstractTransformer instead of {{ARPMH}} and {{BridgeHandler}}. | 4 |
| 564 | XD-638 | Unclear error message using hadoop fs shell commands without configuring a URL | When using the shell, I forgot to configure the Hadoop URL via "hadoop config fs --namenode hdfs://pivhdsne:8020" I then did: xd:>hadoop fs ls /xd/tweets You must set fs URL before run fs commands The error message should tell me how to set the URL (via the hadoop config command). I had to go back to the documentation to figure it out. It's also not entirely grammatically correct. | 1 |
| 565 | XD-639 | Update error message for usage of hadoop rm with --recursive option | I had the following interaction with the shell. It does work if I do "hadoop fs rm /xd/tweets --recursive". Either the order shouldn't matter or the doc should be more clear on placement of the option. xd:>hadoop fs rm /xd/tweets To remove directory, please use fs rm --recursive instead xd:>hadoop fs rm --recursive /xd/tweets java.lang.IllegalArgumentException: Failed to convert '/xd/tweets' to type boolean for option 'recursive' Cannot convert /xd/tweets to type Boolean. | 2 |
| 566 | XD-640 | Cannot start xd-container with the --hadoopDistro option | Trying to use xd-container with PHD, and therefore need to start with --hadoopDistro. I get the following error: $ bin/xd-container --hadoopDistro phd1 17:11:20,305 ERROR main server.ContainerMain:59 - "--hadoopDistro" is not a valid option | 3 |
| 567 | XD-641 | Problem with tapping and > (source channels) | This is a follow on from XD-592. In that bug we fixed up the ability to use tap with pipe. Tap when used as a source channel should also work (and should deploy in a more optimized fashion since source channels can be directly connected to the subsequent module, creation of a pass-through tap instance isn't necessary). This test shows the syntax that should work and the current information about how it fails: {code} public void testTapSourceChannel() throws IOException { FileSink sink1 = newFileSink(); FileSink sink2 = newFileSink(); stream().create("myhttp", "http --port=9314 | transform --expression=payload.toUpperCase() | filter --expression=true > :foo"); // fails with: java.lang.IllegalArgumentException: bean 'myhttp.1' is already // registered but does not match the required type tap().create("wiretap1", "tap myhttp.transform > transform --expression=payload.replaceAll('a','.') | %s", sink1); // fails in TapDefinition ctor with: java.lang.IllegalArgumentException: // streamName cannot be empty or null tap().create("wiretap2", "tap :foo > transform --expression=payload.replaceAll('a','.') | %s", sink2); httpPostData("http://localhost:9314", "Dracarys!"); } {code} I suspect part of the problem initially lies with the code around EnhancedStreamParser that builds the module deployment requests from the Ast parsed from the input DSL string. Whether a source channel was originally specified with 'tap' is captured in that Ast but that knowledge doesn't appear to be getting used. | 4 |
| 568 | XD-642 | Update architecture diagram to show rabbit in addition to redis to communicate between admin and containers | Figure 1 here https://github.com/SpringSource/spring-xd/wiki/Architecture should also show Rabbit as an option, since otherwise people will think we are tied to redis. | 1 |
| 569 | XD-643 | Map column names with underscore to camelCase style keys for JDBC sink | We need to add support for matching column names with underscores like "user_name" and map them to camel case style keys like "userName" in the JdbcMessagePayloadTransformer. | 3 |
| 570 | XD-644 | Connection props in rabbit.properties ignored by xd-admin and xd-container | I modified rabbit.hostname in rabbit.properties and xd-container still attempted to find Rabbit at localhost with --transport rabbit. Looks like the PPC for xd-container and xd-admin is not pointing to rabbit.properties | 2 |
| 571 | XD-645 | Find a way to contribute redis.properties to Rabbit Container PPC | Need to undo the recent add of rabbit.properties to xd-common.xml. Tried to work around this by configuring rabbit-container with only the PPC it needs (pointing to only rabbit.properties), but this caused issues with redis-analytics later requiring redis.properties. Would be nice to have a way for redis-analytics to contribute redis.properties or something similar... Also, strictly speaking, the local admin server does not even need redis.properties, let alone rabbit.properties, so we should find a cleaner way to configure this. | 3 |
| 572 | XD-646 | Split RichGauge in 2 | support of the alpha parameter is awkward and can confuse people who are expecting a simple average mean. Consider splitting RichGauge in two flavors: arithmetic and exponential. Involves quite some work at the repository, handler and REST level though... | 5 |
| 573 | XD-647 | HTTP source should emit raw payload | Current implementation converts to a String. See if we can emit raw payload (given that we also emit content-type header) Setting to 8 points, as this may have lots of implications down the line though | 8 |
| 574 | XD-648 | Regression test on file source | As we overwrote changes to file source by mistake, let's add some regression tests, esp. to the file location. Plan on extending the utility source and sink functionality | 3 |
| 575 | XD-649 | Remove 'substream' from the documentation | This will come back in M3 once we iron out the issues. | 1 |
| 576 | XD-650 | Eclipse build path error after running gradle -> refresh source folders in Eclipse | After running gradle -> refresh source folders on the spring-xd-module project in Eclipse, there is an error because the {{src/test/java}} folder is missing. Solution is to add a placeholder file. | 5 |
| 577 | XD-651 | End point to retrieve a list of all XD artifacts of all kinds | Any kind of sophisticated artifact retrieval mechanism in XD will need to grab more than one kind of artifact at once. For example, if I want to see all taps, streams, triggers, and jobs (ie- everything), I need to make 4 http requests. I can imagine dashboards that need to display information on artifacts of multiple kinds. There will also need to be a way to pass a query to return a sub-set of artifacts, but that should be designed separately. | 5 |
| 578 | XD-652 | File Source Name and Duplicates options not working as documented | The doc says the name option is "the absolute path to the directory to monitor for files" but it actually seems to be the name of a dir in /tmp/xd/input. Not sure which is the correct behavior. Also, "name" as an option name seems a little vague. Maybe something like "--directory"? Also, if I set --duplicates=true, it actually prevents duplicates (setting prevents-duplicates to true) | 2 |
| 579 | XD-653 | Configure Jackson ObjectMappers to Allow Single Quotes | Allow Json payloads from external sources, e.g., http post to contain single quoted field names and values. This is required where XD uses Jackson to convert payloads from Json to object or Tuples. | 1 |
| 580 | XD-655 | Trim output from http post shell command to two lines | Instead of xd:>http post --target http://localhost:9898 --data "hello world" > POST (text/plain;charset=UTF-8) http://localhost:9898 hello world > 200 OK > Content-Length: 0 > Connection: keep-alive > Success sending data 'hello world' to target 'http://localhost:9898' have xd:>http post --target http://localhost:9898 --data "hello world" > POST (text/plain;charset=UTF-8) http://localhost:9898 hello world > 200 OK or better yet xd:>http post --target http://localhost:9898 --data "hello world" > 200 OK POST (text/plain;charset=UTF-8) http://localhost:9898 hello world | 1 |
| 581 | XD-656 | Intra-Module "Pipe" Naming | Consider 2 Admins/Containers that are using discrete Redis instances but a shared Rabbit instance for the transport. If two different streams are deployed on each, but with the same stream name, the Rabbit queues will be common (e.g. foo.0), causing crosstalk. Stream names must be unique across all container instances sharing Rabbit infrastructure. I am not sure what the solution is; two instances of the *same* stream *do* need common queues but instances of different streams need a qualifier of some kind (container name?). I guess it's not *that* big of an issue because, if they're sharing infrastructure, they're likely to be sharing a stream repo too - in which case you'd need unique stream names. | 3 |
| 582 | XD-658 | Update to Spring-Data-Redis 1.1.0.M2 | Remove the {{NoOpRedisSerializer}} and use the non-serialization feature of M2. | 2 |
| 583 | XD-660 | Rename spring-xd-shell to xd-shell | NULL | 1 |
| 584 | XD-661 | Batch Jobs: Add the ability to provide JobParameters | NULL | 4 |
| 585 | XD-662 | No indication of failure in shell when deploying job referencing nonexistent trigger | I see the following output on the shell if I create a job and reference a non-existent trigger. There's a big stack trace in the server log, but nothing on the shell side indicating failure. A subsequent "jobs list" also shows the job. The same thing happens if I deploy an undeployed Job after deleting its associated Trigger. $ job create --name helloWorldJob --definition "myjob --trigger=nonexistenttrigger" Successfully created and deployed job 'helloWorldJob' | 2 |
| 586 | XD-663 | Use correct FS_DEFAULT_NAME_KEY constant based on Hadoop version used | Keep getting the following warning: WARN Spring Shell conf.Configuration:817 - fs.default.name is deprecated. Instead, use fs.defaultFS Should switch to use the runtime value of the FS_DEFAULT_NAME_KEY constant based on Hadoop version used. | 3 |
| 587 | XD-664 | File sink filename should default to having a '.out' suffix. | NULL | 1 |
| 588 | XD-665 | AggregateCounter display command options with "lastHours" and "lastDays" | It would be nice to have "lastHours" and "lastDays" options for aggregatecounter display command. | 1 |
| 589 | XD-666 | Remove Redis Transport Headers from Tapped Stream | Redis transport headers are not removed in taps. | 2 |
| 590 | XD-667 | Add Accepted Media Type Support to Tap | Currently, the initial tap module accepted media types are not retrieved from the module when creating the tap. | 5 |
| 591 | XD-668 | Rest-Client should not force usage of Joda Time | The Rest-Client project should not impose Joda to the user. | 2 |
| 592 | XD-669 | AggregateCounterTemplate should not use Joda | Not only should it not use Joda (see XD-668) but the passing of dates currently relies on default formatting | 3 |
| 593 | XD-670 | TAB completion for existing entities | Provide Shell TAB completion when referencing an existing entity | 4 |
| 594 | XD-671 | Add support for dynamic routing | It should be possible to create streams like the following which rely upon named channel support and dynamic routing capabilities: {code} http | somerouter :x > xtransformer | hdfs :y > ytransformer | hdfs {code} The 'somerouter' processor could return "x" or "y" which determines the downstream path for each message. This should be implemented in such a way that any developer adding a router module would only need to deal with existing Spring Integration semantics (in this case, only considering the return of "x" or "y" - whether it be SpEL or a POJO method invocation). Perhaps in the plugin that modifies a module context, we could simply add a new ChannelResolver implementation (by adding that ChannelResolver as a bean and/or a BeanPostProcessor that configures that as the resolver for any router, if necessary). That ChannelResolver would have a reference to the ChannelRegistry so that the router actually sends its messages to those shared channels. The shared channels themselves would have been created as long as a valid downstream flow has been defined. | 16 |
| 595 | XD-672 | Ugly error messages in shell when not connected | should go thru the list of all commands available and make sure that a simple "not connected" message is returned instead of something like this: {code} org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8080":Unexpected end of file from server; nested exception is java.net.SocketException: Unexpected end of file from server at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:498) .... {code} | 5 |
| 596 | XD-673 | http post in shell incorrectly mentions default of --target option | the value for --target is required (there is no default), but the hint for that option states otherwise: {code} xd:>http post --target http post --target required --target: the location to post to; default if option not present: 'http://localhost:9000' {code} | 1 |
| 597 | XD-674 | Add Spring Batch word-count Sample to Spring XD Samples repo | NULL | 2 |
| 598 | XD-675 | Cannot chain json-field-value-filter & json-field-extractor | Because StringToJsonNodeTransformer expects a String as input, one cannot chain json related processors. A simple solution would be to also accept Jackson IN and forward it directly in that case. | 1 |
| 599 | XD-676 | add PropertyAccessor for Tuple fields in SpEL | Example using name: {code} filter --expression="payload.myfield.startsWith('foo')" {code} Example using index: {code} filter --expression="payload.2.startsWith('foo')" {code} This should support nested keys as well: {code} filter --expression="payload.myfield.subfield.startsWith('foo')" {code} | 5 |
| 600 | XD-677 | Cannot destroy tap if tapped stream is already destroyed | xd:>tap destroy mytap 16:44:41,850 WARN Spring Shell client.RestTemplate:524 - DELETE request for "http://localhost:8080/taps/mytap" resulted in 400 (Bad Request); invoking error handler Command failed org.springframework.xd.rest.client.impl.SpringXDException: XD116E:(pos 4): unrecognized stream reference 'foo' tap foo.http | log Tap is then still listed when I do a "tap list" | 2 |
| 601 | XD-678 | change accepted-media-types to accpted-content-types | NULL | 1 |
| 602 | XD-679 | Update settings file and reformat existing codebase. | Please put in suggestions for the current .settings file. Maybe one suggestion is to not format on save? | 1 |
| 603 | XD-680 | Add Simple Batch Sample to Spring XD Samples repo | This example should require no code. Just the basic XML. | 2 |
| 604 | XD-682 | Modify file sink to avoid dot with empty suffix | The expression currently appends "." + ${suffix} (where the default suffix is 'out'). If the suffix value were an empty String, this would lead to the file name ending with a dot. We should update the expression so that it only appends the dot if the suffix is not empty. This might be possible with a ternary expression. | 1 |
| 605 | XD-683 | Change jmxDisabled option to jmxEnabled and do not enable by default | also, the current behavior is broken; it checks if the property is set but does not actually check whether it's true or false | 2 |
| 606 | XD-685 | Refactor Taps to Avoid Transport Hop | Taps are currently source modules. They could be refactored to simply bridge the tapped module's tap pub/sub topic directly (with conversion) to the first tap module's input channel. Note - ensure destroy works. Currently the tap is destroyed by the simple fact it is a module; if it's no longer a module we'll need special handling to stop/remove the tap adapter. | 16 |
| 607 | XD-686 | Support Named Taps (or Similar) | Provide some syntax allowing multiple tap points to be directed to a named channel. e.g. tap foo.4 > namedTap tap bar.2 > namedTap or :tap.foo > counter | 8 |
| 608 | XD-687 | provide user friendly messages when dealing with invalid gemfire sink | xd:>stream create --name testgemfire --definition "http --port=8887 | gemfire" 16:20:28,503 WARN Spring Shell client.RestTemplate:524 - POST request for "http://localhost:8080/streams" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.xd.rest.client.impl.SpringXDException: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'region' defined in null: Could not resolve placeholder 'regionName' in string value "${regionName}" | 4 |
| 609 | XD-688 | error messages not thrown when creating gemfire sink without starting gemfire server | stream create --name test1 --definition "http --port=8827 | gemfire-server" Created new stream 'test1' stream create --name test2 --definition "http --port=8828 | gemfire-json-server" Created new stream 'test2' even if gemfire server is not started, streams are successfully created. This behavior is inconsistent with hdfs where if hdfs connection is not available, creating stream using 'http | hdfs' will fail. | 4 |
| 610 | XD-689 | Support serialization/deserialization of Message payloads across JVMs across all transports. | String -> byte[] (string.getBytes()) byte[] -> byte[] (no serialization) Pojo -> configured serialization | 1 |
| 611 | XD-690 | Simplify "instance" deployment code | AbstractDeployer has 4 subclasses, 3 of which override e.g. deploy() making the boilerplate factorization ineffective. Introduce an intermediate class for those deployers that support the concept of an instance (Stream, Tap, Job to some extent) | 8 |
| 612 | XD-691 | Change JMX option to reference 'enableJmx' instead of 'disableJmx' | Make the default value of enableJmx false until we have tested/documented JMX functionality | 2 |
| 613 | XD-692 | http source module should copy Content-Type header to SI MessageHeaders.CONTENT_TYPE | NULL | 1 |
| 614 | XD-693 | Add deploy/undeploy commands for taps | NULL | 3 |
| 615 | XD-694 | Need to check the deployment requests in StreamsControllerIntegrationTest | we should check the actual deployment requests were built correctly for each module in the testCreateUndeployAndDeleteOfStream test. Currently we just use the anyListOf check. | 2 |
| 616 | XD-695 | Upgrade SDR to get rid of temporary no-op serializer | Spring Data Redis 1.1 M2 added the ability to use RedisTemplate with binary data. We should switch to that instead of the no-op serializer we were forced to implement previously. | 3 |
| 617 | XD-698 | Shell integration tests should be able to be run across all transports | Automate running integration tests on all supported transports | 10 |
| 618 | XD-699 | Handling tap operations on a tap that has reference to a deleted stream | When trying to undeploy/destroy a tap that has reference to an already deleted stream fails with the following exception: Command failed org.springframework.xd.rest.client.impl.SpringXDException: XD116E:(pos 4): unrecognized stream reference '<stream_name at the tap defintion>'. As expected, the StreamConfigParser's lookupStream fails to find the stream name as the stream doesn't exist in the repository. In this scenario, what is a better way to handle the tap operations. Should we undeploy the tap when the stream is destroyed? ( though I don't see an easy way to find the taps that use a specific stream). | 2 |
| 619 | XD-700 | Remove XD UUIDGenerator in favor of the new SI provided one | Remove <bean id="idGenerator" class="org.springframework.xd.dirt.container.UUIDGenerator" /> (container.xml) Delete org.springframework.xd.dirt.container.UUIDGenerator remove compile dependency on eaio from build.gradle | 1 |
| 620 | XD-701 | Refactor Message conversion in ChannelRegistrySupport | After an initial attempt which was not ready for M2 we are rethinking our strategy. One of the fundamental things we have come to realize is that its important to treat serialization and type conversion as separate concerns. Serialization: A core principle is the consumer should by default receive exactly what the producer sent: - If the producer sends a byte[] payload then no serialization is required. - A String payload can use simple byte conversion, taking the Charset into account - Transporting an Object uses whatever serialization is configured (json, xml, avro, protocol buffer, java.io, msgpack, etc.). The actual serialization performed for each message must be shared with the producer and consumer. I.e., the consumer needs to know which case above applies to each payload. Currently we are using the MessageHeaders.CONTENT_TYPE defining custom mime types for this (The designated header is subject to change) Conversion: - The consumer optionally defines one or more content-types (read MimeType) it can accept in order of preference. If no conversion succeeds, we can either give them the byte[] payload or throw an exception (configurable?). Examples: - Consumer accepts a Java Object (application/x-java-object;type=example.Foo). Assume for simplicity, the consumer may send a JSON String, or a Foo. On the receiving end we need to distinguish a String payload containing a JSON representation of Foo from a serialized Foo payload. If the payload is a String, we need to know that its original content is application/json. We are currently using a 2nd "original-content-type" message header to supply this information. So in the first case we have (conceptually) content-type: "XD plain text" , original-content-type "application/json". In the second case we have content-type: "XD Serialized JSON" original-content-type not used in this case since the serialized JSON includes type information (using Jackson conventions which are a bit brittle). -If the producer type is different from the accepted type, we use the conversion service and the consumer must register appropriate converters. A twist for XD that may be generally relevant is that some optimization is possible when we know the bytes represent JSON: - Tuple conversion: Since we serialize using JSON and we know how to transform JSON <->Tuple, we can convert any Object payload or any JSON String to a Tuple. We can avoid the two step deserialization+conversion, e.g. 1) Foo->JSON->Foo 2) Foo->Tuple. | 10 |
| 621 | XD-702 | stack overflow when trying to create a stream with the same name as a module | (NOTE: even if we do want to prevent the use of module names for stream names, we obviously need to avoid a StackOverflowError) to reproduce: start the xd-singlenode container start the xd-shell, and type the following: {code} xd:>stream create time --definition "time | log" {code} that should produce an Internal Server Error output message check the xd-singlenode console, and find: {code} SEVERE: Servlet.service() for servlet [xd] in context with path [] threw exception [Handler processing failed; nested exception is java.lang.StackOverflowError] with root cause java.lang.StackOverflowError java.lang.StringValue.from(StringValue.java:24) java.lang.String.<init>(String.java:178) org.springframework.xd.dirt.stream.dsl.Token.<init>(Token.java:46) org.springframework.xd.dirt.stream.dsl.Tokenizer.lexIdentifier(Tokenizer.java:195) org.springframework.xd.dirt.stream.dsl.Tokenizer.process(Tokenizer.java:62) org.springframework.xd.dirt.stream.dsl.Tokenizer.<init>(Tokenizer.java:41) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.parse(StreamConfigParser.java:65) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.parse(StreamConfigParser.java:55) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.lookupStream(StreamConfigParser.java:545) org.springframework.xd.dirt.stream.dsl.StreamNode.resolveModuleNodes(StreamNode.java:135) org.springframework.xd.dirt.stream.dsl.StreamNode.resolve(StreamNode.java:122) org.springframework.xd.dirt.stream.dsl.StreamsNode.resolve(StreamsNode.java:53) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.parse(StreamConfigParser.java:74) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.parse(StreamConfigParser.java:55) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.lookupStream(StreamConfigParser.java:545) org.springframework.xd.dirt.stream.dsl.StreamNode.resolveModuleNodes(StreamNode.java:135) org.springframework.xd.dirt.stream.dsl.StreamNode.resolve(StreamNode.java:122) org.springframework.xd.dirt.stream.dsl.StreamsNode.resolve(StreamsNode.java:53) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.parse(StreamConfigParser.java:74) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.parse(StreamConfigParser.java:55) org.springframework.xd.dirt.stream.dsl.StreamConfigParser.lookupStream(StreamConfigParser.java:545) org.springframework.xd.dirt.stream.dsl.StreamNode.resolveModuleNodes(StreamNode.java:135) org.springframework.xd.dirt.stream.dsl.StreamNode.resolve(StreamNode.java:122) org.springframework.xd.dirt.stream.dsl.StreamsNode.resolve(StreamsNode.java:53) ...ad nauseum {code} | 2 |
| 622 | XD-703 | JobRepository should be persistent and shared across xd-admin/xd-container | The current code is creating an in-memory job repository for each batch job that is launched. This makes it impossible to query the tables in the job repository across the cluster. A single job repository that is backed by a file need to be shared across all jobs that are a launched. *Implementation Suggestions* * The XDAdmin server should create the job repository schema, if not found, in a HSQLDB database, when it starts up * The bean definitions should be added to the same context that the analytics are being loaded in as it is already shared across xd-admin/xd-container. The ‘analytics’ context should be renamed to something more generic, ‘shared parent context’ or something. * There is some clean up (removal) of the code in the current JobPlugin, META-INF/spring-xd/plugins/job/common.xml wouldn’t be needed anymore. That might be all, not sure. *How to verify it works* * A JUnit test that verifies the spring batch tables were created in the job repository when xd-admin is launched. This would require deleting the backing db file before instantiating singlenode/xd-admin/xd-container. * If you start the xd-container it should be able to find the necessary DataSource/JobRepository beans information to be able to contact the database. We don’t have DI style JUnit tests so this will required getting a reference to the xd-container and it’s application context, and performing ‘getBean(JobRepository.class)’ | 8 |
| 623 | XD-704 | User should be able to specify Rabbit virtual host | Need to support Rabbit virtual host property in properties file and as args to Rabbit source and sink | 2 |
| 624 | XD-705 | A batch job can be launched by sending a message on a channel | *Description* When a job is ‘created’ in SpringXD, a ‘control-channel’ for that job is also created. The listener for that channel will receive a message, be able to take the ‘jobParameters’ and other launch information from the message, and be able to launch/run the job. NOTE: I can see a few other stories that should probably be made to break this up after writing it. I put estimate of 10 for now, but we should break this up. Here are some suggestions 1. create ‘data only’ JobParametersBean equivalent with primitive types 2. create jobLauncher source 3. create jobParameterTransformer processor 4. Refactoring of ChannelRegistry’s aliasHing to use a callback strategy. *Implementation Suggestions* job create --name helloWorldJob --definition "myjob --somePropertyToOverride=someValue * This would not execute the batch job immediately, but instead register the job definition and deploys a “jobLauncher�? and the job definition to an XD-Container. * The XD-Container that receives the deploy request message will create a module application context, will also create a channel with the Channel registry named after the job, e.g. :myJob. This should be a pub/sub channel from the point of view of the middleware. From the point of view of the spring integration channel, it should ideally be of the executor channel. There is a limitation in the current implementation of ChannelRegistry now as ‘createInbound’ only creates direct channels. The boolean ‘aliasHint’ should probably be extended to some type of callback that creates a channel. The aliasHint was added to address the case of LocalChannelRegistry creating or looking up a queue backed channel or a direct channel. There will be a consumer on the SI channel in the module application context that will be responsible for getting the job launch information and launching the job. The launching of the job may need to be explicitly done in a separate thread if direct channels are created by the ChannelRegistry. The contents of the message should be something similar to the current “JobParametersBean�?, it needs to be easily serializable with simple types via JSON over the wire. The current impl of “JobParametersBean�? has ObjectMapper, so that may require a bit of reworking. The handler of the message will use the jobLauncher to launch the job, using the information in the JobParametersBean. * The ‘myjob’ can then be launched by sending a message, perhaps this is handled by having a jobLauncher source jobLauncher [--jobParameters <jobParameters>] [--dateFormat <dateFormat>] [--numberFormat <numberFormat>] [--makeUnique <makeUnique>] > :myJob e.g. with no-args jobLauncher > :myJob *How to verify it works* With the test HelloSpringXDTasklet, we should be able to create the job job create --name helloSpringXD --definition "myjob" This will not launch the job (as mentioned in the ‘implementation’ section. It would then be launched by jobLauncher > :myJob where jobLauncher is a new source. Ideally would like to be able to test a data driven triggering. This would require a new file source that doesn’t use the file-to-string-transformer, but lets a File object be the payload. file | jobParameterTransformer > :myJob | 16 |
| 625 | XD-707 | Support use of separate control and message transports | Control transport - Deploy/Undeploy requests Message transport - Inter module communication Currently complicated because starting a Job for example currently uses message transport vs control transport. Testing scenarios require local control and ability to switch to various message transports. One option is to change the interpretation of transport command line arg depending on SingleNode, Admin, or Container. e.g. SingleNode --transport rabbit (always use local for control messages) Admin (requires --transport, message transport does not apply) Container (enforces the same transport for message and control. Local optimization done via composite module) The other option is use a separate transport for control vs messages. Either way need to rationalize the design with respect to control and module messages | 2 |
| 626 | XD-708 | Add SingleNodeMain class | SingleNodeMain(){ parent = new AC(..) AdminMain.launch(parent); ContainerMain.launch(parent); } This should make startup processing more consistent and symmetrical | 3 |
| 627 | XD-713 | Support for @Configuration based module definitions | NULL | 3 |
| 628 | XD-715 | Upgrade Lettuce and Netty | Upgrade Lettuce to 2.3.3 and subsequently Netty to 3.6.6 | 2 |
| 629 | XD-716 | TapCommandTests hangs when using a lazily instantiated Lettuce connection | A change was made in spring-data-redis to instantiate the shared Lettuce connection lazily instead of when the context is initialized. This caused TapCommandTests to hang due to a Netty worker thread trying to initialize the Lettuce connection (Lettuce uses Netty). The change was temporarily backed out of SDR, but we need to consider using a NettyExecutionHandler in NettyHttpInboundChannelAdapter or making the HTTP module's "input" channel an ExecutorChannel to avoid potentially long operations like from happening in an I/O thread. Also, we need to address why this failure simply hangs the shell. Shell was hung waiting on IO here: org.springframework.http.client.SimpleClientHttpResponse.getRawStatusCode(SimpleClientHttpResponse.java:47) org.springframework.http.client.AbstractClientHttpResponse.getStatusCode(AbstractClientHttpResponse.java:32) org.springframework.xd.shell.command.HttpCommands$1.hasError(HttpCommands.java:93) org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:484) org.springframework.web.client.RestTemplate.execute(RestTemplate.java:460) org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:335) org.springframework.xd.shell.command.HttpCommands.postHttp(HttpCommands.java:103) sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:191) org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:64) ringframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:57) - locked <7fd3c7d40> (a java.lang.Class for org.springframework.shell.core.SimpleExecutionStrategy) org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:127) org.springframework.xd.shell.AbstractShellIntegrationTest.executeCommand(AbstractShellIntegrationTest.java:99) org.springframework.xd.shell.AbstractShellIntegrationTest.httpPostData(AbstractShellIntegrationTest.java:112) org.springframework.xd.shell.command.TapCommandTests.testCreateAndDeployTap(TapCommandTests.java:56) Full stack trace of server exception: Aug 19, 2013 9:59:00 AM org.jboss.netty.channel.SimpleChannelUpstreamHandler WARNING: EXCEPTION, please implement org.springframework.integration.x.http.NettyHttpInboundChannelAdapter$Handler.exceptionCaught() for proper handling. org.springframework.integration.MessageHandlingException: org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis on localhost:6379; nested exception is com.lambdaworks.redis.RedisException: Unable to connect org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:76) org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:67) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:137) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.BroadcastingDispatcher.invokeHandler(BroadcastingDispatcher.java:121) org.springframework.integration.dispatcher.BroadcastingDispatcher.dispatch(BroadcastingDispatcher.java:112) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.interceptor.WireTap.preSend(WireTap.java:121) org.springframework.integration.channel.AbstractMessageChannel$ChannelInterceptorList.preSend(AbstractMessageChannel.java:248) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:173) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:92) org.springframework.integration.x.http.NettyHttpInboundChannelAdapter.access$200(NettyHttpInboundChannelAdapter.java:59) org.springframework.integration.x.http.NettyHttpInboundChannelAdapter$Handler.messageReceived(NettyHttpInboundChannelAdapter.java:122) org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:81) org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:485) org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) Caused by: org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis on localhost:6379; nested exception is com.lambdaworks.redis.RedisException: Unable to connect org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.createLettuceConnector(LettuceConnectionFactory.java:345) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.initConnection(LettuceConnectionFactory.java:116) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:325) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:106) org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:81) org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:53) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:173) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153) org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86) org.springframework.data.redis.core.DefaultZSetOperations.add(DefaultZSetOperations.java:41) org.springframework.data.redis.core.DefaultBoundZSetOperations.add(DefaultBoundZSetOperations.java:47) org.springframework.xd.store.AbstractRedisRepository.trackMembership(AbstractRedisRepository.java:202) org.springframework.xd.analytics.metrics.redis.RedisCounterRepository.increment(RedisCounterRepository.java:88) org.springframework.xd.analytics.metrics.redis.RedisCounterRepository.increment(RedisCounterRepository.java:82) org.springframework.xd.analytics.metrics.integration.MessageCounterHandler.process(MessageCounterHandler.java:28) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69) org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:97) org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:82) org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102) org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:103) org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:126) org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:230) org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:129) org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:73) ... 84 more Caused by: com.lambdaworks.redis.RedisException: Unable to connect com.lambdaworks.redis.RedisClient.connect(RedisClient.java:176) com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:139) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.createLettuceConnector(LettuceConnectionFactory.java:339) ... 111 more Caused by: java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread. org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:342) org.jboss.netty.channel.DefaultChannelFuture.await(DefaultChannelFuture.java:231) com.lambdaworks.redis.RedisClient.connect(RedisClient.java:166) ... 113 more | 5 |
| 630 | XD-717 | cat command doesn't work when same data is listed in file multiple times | $ ./xd-admin --transport redis $ ./xd-container --transport redis $ ./xd-container --transport redis xd:>stream create --name httpStream --definition "http | file" Created new stream 'httpStream' xd:>tap create --name httpTap --definition "tap httpStream | counter" Created and deployed new tap 'httpTap' xd:>http post --target http://localhost:9000 --data "helloworld" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld > 200 OK xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld xd:>http post --target http://localhost:9000 --data "helloworld" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld > 200 OK xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld xd:>http post --target http://localhost:9000 --data "helloworld2" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld2 > 200 OK xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld helloworld2 xd:>http post --target http://localhost:9000 --data "helloworld3" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld3 > 200 OK xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld helloworld2 helloworld3 xd:>http post --target http://localhost:9000 --data "helloworld3" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld3 > 200 OK xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld helloworld2 helloworld3 xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld helloworld2 helloworld3 xd:>http post --target http://localhost:9000 --data "helloworld3" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld3 > 200 OK xd:>http post --target http://localhost:9000 --data "helloworld3" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld3 > 200 OK xd:>http post --target http://localhost:9000 --data "helloworld3" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld3 > 200 OK xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld helloworld2 helloworld3 xd:>http post --target http://localhost:9000 --data "helloworld4" > POST (text/plain;Charset=UTF-8) http://localhost:9000 helloworld4 > 200 OK xd:>! cat /tmp/xd/output/httpStream.out command is:cat /tmp/xd/output/httpStream.out helloworld helloworld2 helloworld3 helloworld4 xd:>counter display --name httpTap 9 xd:> however in the regular shell. $ cat /tmp/xd/output/httpStream.out helloworld helloworld helloworld2 helloworld3 helloworld3 helloworld3 helloworld3 helloworld3 helloworld4 | 2 |
| 631 | XD-718 | Use ParentLastClassLoader to create the Modules ApplicationContext. | The ParentLastClassloader is located in the spring-hadoop project. It will resolve classes first looking at the child context and then the parent. This works well for XD since the we want and dependencies of the module to be considered first and if not found, resolve against the parent. It would be possible to even include other versions of .jars already in the parent classloaders (e.g. Spring Integration jars), but for now, we will not immediately test that case. SimpleModule needs to change to that we can pass in the classloader to use for creating the application context. The current implementation creates a GenericApplicationContext as a field initializer...that should change to be in the ctor. ModuleDeployer should implement .BeanClassLoaderAware. The classloader passed in to the BeanClassLoaderAware callback will be used as the ‘parent’ when creating the ParentLastClassloader. The URL[] to pass into ParentLastClassloader should be ‘null’ or an empty array in this case of an older style module. (Hopefully ParentLastClassloader allows that type of fallback). *Implementation Suggestions:* The ModuleDeployer code is where the application context for the module is defined and instantiated. Here is a possible impl path. 1. Assuming we can always use ParentLastClassloader (even for older style modules), then the ModuleDescriptor needs to return an array of URL[] locations for the module, getURL(). This is passed into the cto for SimpleModule. The ctor then creates a new application context, creates the parentclassloader, sets the classloader on the application context and then proceeds as normal. 2.AbstractModuleRegistry should try and load the resource from two possible locations, e.g. ./modules/source/file/config/file.xml or ./modules/source/file.xml The module registry needs to be a bit smarter to know, ah, i see a config directory, let me try ./config/file.xml otherwise just ./file.xml *How to verify it works.* 1. JUnit test in which one of the ModuleRegistry implementations points to a test directory that contains both old and new style modules. FileModuleRegistry is probably a good choice here. Need to test that the new getting for URL[] works as expected. 2. Existing tests should run as they did before, in particular the shell integration tests. | 5 |
| 632 | XD-719 | Refactor the file source module to have lib/config directories | Convert a simple module, such as file, to further test that what was done in the previous story, “Use ParentLastClassLoader to create the Modules ApplicationContext.�? works as expected. *Implementation Suggestions* Remove from build.gradle the dependency on spring-integration-file and place that jar inside a directory ./modules/source/file/lib place the current file.xml inside ./modules/source/file/config *How to verify it works.* 1. Running tests that currently use the file source, e.g. in spring shell, should work as before 2. When deploying a stream file | log, we should be able to interrogate the channel registry and make sure it found the dependencies for the module, ModuleDescriptor should have a not-null URL[] property. | 5 |
| 633 | XD-720 | Deploy a new source module *after* XD-singlenode container has started. | After the xd-singlenode process has started, create a new module that has dependencies not already in the parent application context, and then create a stream that uses the new module. *Implementation Suggestions* Develop in the test tree, maybe of the xd-shell project, a new module. the lib and config should be sititng around in a directory waiting to be copied into the appropriate spot. Could try http://www.date4j.net/..The config file would be similar to time.xml but use the date4j class. *How to verify it works* In a JUnit test case copy in a new module that has new dependencies, ..copy the lib and config directories into the location where the ModuleRegisty will pick it up. Deploy a new stream, date4j | file and see if there are contents in the file. | 5 |
| 634 | XD-721 | Deploy a new job module *after* XD-singlenode container has started. | After the xd-singlenode process has started, create a new job that has dependencies not already in the parent application context, and then create and run a new job that uses the new module. *Implementation Suggestions* Develop in the test tree, we can put in the jar and config from https://github.com/SpringSource/spring-xd-samples/tree/master/batch-simple *How to verify it works.* In a JUnit test case copy in a new job that has new dependencies, ..copy the lib and config directories into the location where the ModuleRegisty will pick it up. Deploy a new job and check in the job repository that the job ran and was successful. | 5 |
| 635 | XD-722 | Batch jobs send job and step events on channels | This is the other side of launching a job by sending a message. The Job plugin should add listeners at the job/step level so that the job/step context information can be sent out on a channel. :myjob.notifications is a suggested channel name that would be created automatically. | 8 |
| 636 | XD-723 | Change inconditionnal Thread.sleep() calls in tests to smarter incremental pauses | There are a lot of Thread.sleep() calls with delays chosen in the 1-2 seconds range. Change to a while loop with smaller pauses until a timeout is reached and give up. This applies to verification code (e.g. verifying that a counter has expected value) as well as File setup, or http being ready to accept requests etc | 8 |
| 637 | XD-724 | Test source module in isolation | Register the module under test and deploy the module Verify output across all transports Examples Be able to start the rabbitmq source just by pointing to modules/source/rabbit.xml, pass in some property file for parameters to be replaced, and outgoing message is placed in a in-memory queue backed channel for use with assertions to verify functionality. Test that sending json, results in media-type header is set to json Test that sending POJO -> POJO Test that sending Tuple -> Tuple Test that sending a (JSON) String -> String Test that sending raw bytes -> raw bytes | 5 |
| 638 | XD-725 | Test processor module in isolation | Register the module under test and have access to a source channel that drives messages into the processor and a output channel where output messages are sent. Examples Built-in Message conversion: send JSON to a processor module that accepts Tuples. | 5 |
| 639 | XD-726 | Test sink module in isolation | Register the module under test Send a message to the sink using a test source and verify the sink contents - this requires checking an external resource - depends on the sink | 5 |
| 640 | XD-727 | Create directory structures and move existing UI code into Spring XD repository | Create a directory structure that best benefits UI development. The copying of the UI files and other gradle build tasks so that the UI can be run inside the embedded servlet container of XD will be a seperate story | 4 |
| 641 | XD-728 | Display the UI from xd-admin container when doing development in eclipse | The UI code will be sitting in one or more top level directories in the repository This story will address the need to 1) copy over the UI code into a location so that it can be picked up by the embedded servlet container when running inside eclipse | 2 |
| 642 | XD-729 | Package up the UI code when building the distribution so that it can be shown by xd-admin | The UI code will be sitting in one or more top level directories in the repository This story will address the need to 1) copy over the UI code into a location so that it can be picked up by the embedded servlet container when the distribution zip is 'unzipped'. After running ./xd-admin or ./xd-singlenode one should be able to hit the UI at http://localhost:8999/xd (just an example) | 2 |
| 643 | XD-730 | Add additional embedded servlet container config to load static UI resources | Configure embedded servlet container needs to know where to load the UI code. | 2 |
| 644 | XD-731 | Replace 'anonymous' node in XD module bean names | Enhance bean naming strategy or provide a value for the property that binds to this | 1 |
| 645 | XD-732 | Add PropertyAccessor for JSON fields in SpEL | {code} filter --expression="payload.myfield.startsWith('foo')" {code} Example using index: {code} filter --expression="payload.2.startsWith('foo')" {code} This should support nested keys as well: {code} filter --expression="payload.myfield.subfield.startsWith('foo')" {code} This is related to https://jira.springsource.org/browse/XD-676 and that in turn depends on SI being able to configure SpEL | 4 |
| 646 | XD-733 | Create JobLaunchRequest Transformer | The JobLaunchRequest Transformer shall accept the following payloads: * File * JSON String * Properties * Map * Tuple Use/Migrate some of the logic from *JobParametersBean*, e.g. using the *DefaultJobParametersConverter*. *Special Case File* When handling a *File*, add special JobParameter *absoluteFilePath* populating it with *message.getPayload().getAbsolutePath()* * Add unit tests | 6 |
| 647 | XD-734 | A job will be associated with a named channel when the job is created. | When creating a job, a named channel will be created with a name of job.<your job name> i.e. job.foo. ======== Required components: - A Transform See XD-733 - JobPlugin needs to create the NamedChannel for the job and associate the transform. > Registrar.xml will need an input channel? > Name channel support will be required. - Add --channel to job rest apis to notify system that a named channel is requested. *Unit Test | 5 |
| 648 | XD-735 | Job Repo in container needs access to Admin's HSQLDB | The HSQLDB that stores the JobRepository needs to have its content exposed via TCP (network service) so that container has access to update the status of a job run. -Needs to have property that enumerates the host and port for the admin that is accessible by the user. -If one is not specified it should default to localhost:9500. *Testing: - Unit Tests - Bring up module and admin. ^Verify that default host and port work ^Verify that container on different machine has access to admin | 5 |
| 649 | XD-736 | Expose restful services that allow users to view job statuses | This story utilizes BatchAdmin and its restful interfaces to show the state of jobs in Spring XD. *Steps - Create a Branch in the BatchAdmin (we don't want to lose history) - Update the restful API's to XD standards. - Create bamboo task to push jars to artifactory - Update gradle.build to pull in the Batchadmin jars. - Expose the restful calls. | 5 |
| 650 | XD-737 | Trigger can send a message to a named channel | Trigger can send a message to a named channel. For example: trigger create --name mytrigger --definition "trigger --cron='*/10 * * * * *' --message='Good Luck, we are all counting on you'" --channel foo Where the --message contains the message that will be sent to foo job/component. | 3 |
| 651 | XD-738 | Rename XDContainer and associated classes to Node | Rename XDContainer, ContainerMain, *ContainerLauncher, ContainerLauncherFactory and any variable or methodNames, bean names, etc. that refer to container in favor of the term 'Node'. Eliminate the dirt.container package, and move Node into .server | 2 |
| 652 | XD-739 | Eliminate internal dependencies on System properties | Remove System.setProperty() or System.getProperty() for internal xd properties. Use spring Environment abstraction instead. Also, replace "." in property names with '_' (XDPropertyKeys). This is compatible with environment variable names. As a result, XD should accept System properties or environment variables or command line options. Command line options should have highest precedence. Retain StandardEnvironment order wrt to System properties and environment variable. | 4 |
| 653 | XD-740 | Remove Option parsing code used for tests from Servers | Currently *Main class provide alternate static methods for parsing CLI options. One used for testing does not call System.exit() just throws an exception. This code should be moved to spring-xd-test to support integration testing. | 3 |
| 654 | XD-741 | Rename xd-global-beans.xml | The above config contains beans that must be in a common parent context for the AdminServer and Modules. Hence not really global since the (Node) doesn't need them itself. So the name is a bit misleading. Come up with something better. | 2 |
| 655 | XD-742 | REST API for Job Management | Spring Batch Admin provides a complete, but outdated implementation style, which covers the full administrative lifecycle of batch jobs, their creation, stop/start, and retrieving information about previous job executions and the status of currently executing job executions. SpringXD has a different way of deploying, starting, and stopping jobs - by sending messages to containers that run the batch job. However, the reporting state of a job is still stored in a common job repository. The purpose of this story is to take the first step to merging in the existing code base that focuses only on the retrieval of information from Spring Batch Admin’s Job controller. The current ‘REST API’ style of these commands should stay as close to the original spring batch admin style as possible. There are several reasons for this 1. It works, and time to springone is short, and we mgmt has expectations around deliverables that we must strive to meet. 2. It gives Andrew a working contract to start developing a UI 3. We can take on this technical debt, but refactor after RC1 and before GA while and deliver end-user functionality. Attached is the list of endpoints in spring batch admin | 20 |
| 656 | XD-743 | Ensure that when batch jobs are created, they are created with the job bean definition id equal to the ‘stream name’ | Unlike in spring-batch-admin, in SpringXD all the jobs the /modules/jobs directory is not ‘visible’ to query when the server starts. Jobs only become visible to XD’s ‘jobs list’ command once they have been ‘created’. Creating a Job in XD is an opportunity to specify additional values to any property placeholders in the job bean definition. This isn’t part of spring-batch-admin. We will not worry about the creation of job definition in this story. Assume that they have been created already and that the GET for /jobs works as it does now for Spring XD. We should however, make sure that there is always a replacement of the job name in the job bean definition to match the ‘--name’ specified in the command line. That is “job create --name myjob --description “thisfunkyjob�? will use ‘myjob to replace <job id="${xd.stream.name}" in the file thisfunkyjob.xml *Implementation Suggestions* This should hopefully just be a matter of changing job definition files to follow the naming pattern. <job id="${xd.stream.name}" … /> *How to verify it works* 1. Create a JUnit integration style test that has ‘job create --name myjob --defintion “testJob�?’ and then deploy the job. The name ‘myjob’ should appear in the job execution table | 1 |
| 657 | XD-744 | Add dependency to spring batch admin in spring-xd-dirt | We should depend on <dependency> <groupId>org.springframework.batch</groupId> <artifactId>spring-batch-admin-manager</artifactId> <version>${project.parent.version}</version> </dependency> <dependency> <groupId>org.springframework.batch</groupId> <artifactId>spring-batch-admin-resources</artifactId> <version>${project.parent.version}</version> </dependency> we are using spring batch 2.2.0.RELEASE. We need to depend on spring-batch-admin version 1.3.0.BUILD-SNAPSHOT | 2 |
| 658 | XD-745 | Return the list of Jobs from spring-batch-admin | The current XD JobController that returns a list of jobs has quite a different API signature than what is in spring-batch-admin. To simplify the UI development, a new controller JobAdminController, will be created that lives under the request path /jobs/admin. The goal is to return a the current JSON structure of spring-batch-admin /jobs/ request and make only minimal changes to implementations of controllers as found . See #1 in the Doc (link to json output doc for spring batch). *Implementation Suggestions* There will need to be some SpringMVC setup that will enable the current style of spring-batch-admin controller requests to co-exist with the existing XD Controllers, e.g. the use of .json for json marshalling etc. This may in fact be the bulk of time spend in this first story to integration spring-batch-admin style controllers into XD. A new controller named JobAdminController that in the spring-xd-dirt project in the package org.springframework.xd.dirt.rest. The JobAdminController will not need to follow the same HATEOAS style as the other controllers at this time. The current controller in Spring Batch Admin looks like this {code} @RequestMapping(value = "/jobs", method = RequestMethod.GET) public void jobs(ModelMap model, @RequestParam(defaultValue = "0") int startJob, @RequestParam(defaultValue = "20") int pageSize) { int total = jobService.countJobs(); TableUtils.addPagination(model, total, startJob, pageSize, "Job"); Collection<String> names = jobService.listJobs(startJob, pageSize); List<JobInfo> jobs = new ArrayList<JobInfo>(); for (String name : names) { int count = 0; try { count = jobService.countJobExecutionsForJob(name); } catch (NoSuchJobException e) { // shouldn't happen } boolean launchable = jobService.isLaunchable(name); boolean incrementable = jobService.isIncrementable(name); jobs.add(new JobInfo(name, count, null, launchable, incrementable)); } model.addAttribute("jobs", jobs); } {code} Something like {code} @RequestMapping(value = "/jobs/admin/jobs", method = RequestMethod.GET) public void jobs(ModelMap model, @RequestParam(defaultValue = "0") int startJob, @RequestParam(defaultValue = "20") int pageSize) { // We do *not* have to query the Spring Batch Admin “JobService�? at this time, but // instead use the JobDeployer to get information about jobs launched by Spring XD Iterable<JobDefinition> jobDefinitions = dobDeployer.findAll() // copy these over to a List<JobInfo> as best as possible, copy name over. // not sure how ‘description’ is getting added to the JSON // pari } {code} *How to verify it works* A sample job needs to be in the modules/job directory. JobCommandTests/AbstractJobIntegrationTest seems to have what is need to stage a job for | 5 |
| 659 | XD-746 | Gradle Launch needs to use singlenodemain vs. admin main | NULL | 1 |
| 660 | XD-747 | Bootstrap XD on Yarn | 1. How XD Yarn application should be packaged and bootstrapped? 2. Where the code should be? Within xd itself or separate repo? | 1 |
| 661 | XD-748 | Interacting with XD on Yarn | 1. How we talk to the XD instance(s) on Yarn 2. There is a rest interface which location can be exposed either via resource manager or appmaster 3. Technically appmaster could also expose interface which could eihter be proxy for xd rest or dedicated interface implementation(i.e. thrift or spring int) | 1 |
| 662 | XD-749 | Comm protocol for appmaster | We need to be able to talk to appmaster which will control the whole xd yarn app. 1. Choose the implementation? Thrift? Spring Int? Something else? | 1 |
| 663 | XD-750 | Container and Grid Control | 1. We'll need a system which give better control of what yarn/xd containers are out there and what is a status of those containers. 2. We also need grouping of containers order to choose, prioritize and scale tasks. 3. We need heartbeating of the grid nodes. Hadoop Yarn itself doesn't give enough tools to know if container is "alive". | 1 |
| 664 | XD-751 | XD UI on Yarn | Technically speaking of we want to integrate XD UI on Hadoop tools we should do it so that the proxy on resource manager works with XD UI. From Hadoop Yarn resource manager point of view this proxied url is the applications tracking url(which is registered when application is deployed). | 1 |
| 665 | XD-752 | Restrict Job launcher with more than one batch job configured in job module | Currently the Job launcher launches all the batch jobs configured in the job module. Please refer, ModuleJobLauncher's executeBatchJob(). This makes the JobRegistry registers with multiple batch jobs under the same Spring XD job name (group name). Also, it is understood that having multiple jobs configuration under the same config xml is uncommon. | 2 |
| 666 | XD-753 | JDBC property settings need to be made externally configurable | We need to have a properties section (documented as well) so that users can setup their jdbc connections for the various components. | 2 |
| 667 | XD-754 | Fix Class/Package Tangle Introduced by XD-353 | {{container}} and {{event}}. {{XDContainer}} references and is referenced by {{ContainerStartedEvent}} (and stopped). https://sonar.springsource.org/drilldown/measures/7173?metric=package_cycles&rids%5B%5D=7717 | 1 |
| 668 | XD-755 | Reactor Environment Improvements | Use a profile or similar to only include the {{Environment}} conditionally (currently in module-common.xml. Also Jon Brisbin one thing to keep in mind: we talked about having a properties file for XD that configured the RingBuffer et al in a non-default way Jon Brisbin e.g. no event loop Dispatchers…a ThreadPoolDispatcher with a large thread pool size (50 threads? 100?)…and maybe even two RingBufferDispatchers: input and output Jon Brisbin so we might want to change from strictly a default Environment bean to an EnvironmentFactoryBean with a specific configuration…thinking about it now I maybe should add a namespace element for the Environment | 3 |
| 669 | XD-756 | Create a splitter module | The splitter functionality in Spring Integration should be exposed to XD as a processing module. The splitter should use a SpEL expression to specify how to split the message up. *Implementation Suggestions* This should be a simple XML based module definition that has input/output channels and has the SpEL expression parameterized. The default value of the SpEL expression should result in the message not being split. *How to check it works* The current file or tail input source can be used to split up the text in a file into words. The tail module should be checked to see how many lines of text it will read into memory at once. The file module node with the file-to-string transformer will only work for small files as it keep the whole file in memory. If there is a big memory inefficiency in using the tail file input source, create a new story to investigate how to have a file based input source that creates a message per line of text or something that is will not result in excessive memory usage. | 2 |
| 670 | XD-757 | Create aggregator module | Create a processing module based on SI's aggregator component. The completion criteria for the aggregator should be a simple count of messages (e.g. received 50 messages) and a timeout so that messages don't stay in the aggregator module for more than 30 seconds. *Implementation suggestions* Create an XML based processing module definition using the SI aggregator namespace. Only the options to support the features in the description should be exposed as property placeholders. *How to know it works* A shell style integration test that has a source that sends a known amount of messages. A ticktock like module would perhaps be a good example. 10 messages sent every 100ms with an aggregator set to a 'aggregate count' of 10, should have 1 message output (perhaps to file sink whose name is based on time as well, is that possible.). A ticktock example with a 1 second delay and with the aggregator module set to have a timeout of 0.5 seconds will have only one message in the file (again assumign the file name has a timestamp/counter in the filename). | 3 |
| 671 | XD-758 | Create Syslog -> Tuple Reactor Codec; Change UDP Syslog Adapter to Emit a Tuple | UDP and Legacy syslog sources emit a {{Map}}; reactor emits a POJO. Make them consistent and emit {{Tuple}}s. | 2 |
| 672 | XD-759 | The xd-singlenode script should have execute permissions | The xd-singlenode script currently has '644' permissions unlike xd-admin and xd-container (which have '755'): {code} -rwxr-xr-x 1 mark staff 5899 Aug 26 16:19 xd-admin -rwxr-xr-x 1 mark staff 5955 Aug 26 16:19 xd-container -rw-r--r-- 1 mark staff 5919 Aug 26 16:19 xd-singlenode {code} | 1 |
| 673 | XD-760 | Fix Package Tangle between o.s.xd.dirt.event and o.s.xd.dirt.container | https://sonar.springsource.org/drilldown/measures/7173?metric=package_cycles&rids%5B%5D=7717 | 3 |
| 674 | XD-761 | Make Spring XD buildable with Java 8 | JavaDoc issues are causing the build to fail with Java 8 | 8 |
| 675 | XD-762 | Add Spring XD Build Plan for Java 8 to Bamboo | This is issue depends on XD-761 https://build.springsource.org/browse/XD-JDK8 | 2 |
| 676 | XD-763 | Remove Trigger Module Code | Triggers will be a source and no longer as a unique module. * The following have to be removed: ** spring-xd-dirt: -- package: org.springframework.xd.dirt.plugins.trigger -- META-INF: spring-xd/plugins/triggers.xml -- org.springframework.xd.dirt.stream.TriggerPlugin *The following beans will require updates to remove the trigger code ** spring-xd-dirt: --META-INF: spring-xd/internal/deployers.xml - Remove Triggerdeployer --org.springframework.xd.dirt.plugins.job.JobPlugin - Remove the registrars for fixedDelay, fixedRate, Cron. As well as the component selection, only need the job-modules-bean --Update the tests to use the trigger as a source, instead of the trigger module. ** spring-xd-shell: Remove trigger commands and associated tests ** xd controllers: Remove trigger controllers and their associated tests This list cover most but not all the components affected. --Success criteria-- Successful unit and integration tests. | 5 |
| 677 | XD-764 | Consolidate Trigger Sources into a single Source | Currently we have 2 trigger sources: trigger & cron-trigger. The preference is to have a user to just use a single trigger source. for example: * trigger > :myjob * trigger --cron='...' >:myjob * trigger --fixedDelay='...' > :myjob One option to handle this is to use spel to reference a bean and then have different trigger beans defined. i.e. trigger='cronTriggerBean'. Each trigger bean would setup the channel with the correct poller. | 5 |
| 678 | XD-765 | Remove AutoLaunch feature from batch jobs | Jobs will be started via trigger. So we won't need the JobTriggerBean. | 1 |
| 679 | XD-766 | Parser needs to handle a ':' embedded in a name. | Also drop the enhanced portion of the EnhancedStreamParser. | 8 |
| 680 | XD-767 | Add Email Source | NULL | 8 |
| 681 | XD-768 | Add Email Sink | NULL | 8 |
| 682 | XD-769 | Add XMPP Sink | NULL | 5 |
| 683 | XD-770 | Update Batch Job docs to cover triggers as a source | NULL | 2 |
| 684 | XD-771 | Shell: Add named channel list command | User shall have the ability to get a listing of available named channels (order by name ascending) from the shell * Add support to controllers * Add tests | 10 |
| 685 | XD-772 | Factor out duplicated SpEL / script logic | See discussion at https://github.com/SpringSource/spring-xd/pull/250/files#r6034885 | 3 |
| 686 | XD-773 | Tab support inconsistent for http post | When doing *xd:> http post* and press the *tab* key. One should get a list of available options. Right now nothing happens. I have to press *--* and then tab to get the options. Interestingly, this works for *stream create* + *tab* key | 2 |
| 687 | XD-774 | Need to create a Persistent-Job-Registry | In order to hook up the to get access to all the jobs available the job registry has to be shared. currently the only implmentation is is the MapJobRegistry. ==== Testability. ==== The admin will need to be see all jobs created by its containers. | 2 |
| 688 | XD-775 | Document router processor module | for an example, see comments here: https://jira.springsource.org/browse/XD-671 | 2 |
| 689 | XD-776 | Shell: Remove "taps list" command | We should only allow "tap list" - currently "tap list" AND "taps list" are allowed but "tap list" does not show up under help. | 1 |
| 690 | XD-777 | Add validation on tap definitions that checks for module names that are part of the stream definition | Try: {code} stream create --name aa --definition "time | log" tap create --name t1 --definition "tap aa.log | log" {code} Results in: {code} Command failed org.springframework.xd.rest.client.impl.SpringXDException: java.lang.NullPointerException {code} | 2 |
| 691 | XD-778 | ${xd.home}/data shows up after gradlew build | When the XD starts up and does not see its job repo it will create one in the${xd.home}/data directory. When xd.home is not set in system properties the job repo creates a literal ${xd.home}/data directory. | 2 |
| 692 | XD-779 | Infinite recursion (StackOverflowError) when trying to process JobLaunchingMessageHandler's "notifications" channel output | Transport used: Redis It looks like when the job is launched, the RedisChannelRegistry's composite handler tries to transform the JobLaunchingMessageHandler's output-channel (notifications) payload which is of type "org.springframework.batch.core.JobExecution". and, This results in Infinite recursion (StackOverflowError). Please see the stack trace here: 01:09:44,827 ERROR task-scheduler-1 redis.RedisQueueInboundChannelAdapter:148 - Error sending message org.springframework.integration.MessageHandlingException: org.springframework.context.ApplicationContextException: Failed to start bean 'jobTriggerBean'; nested exception is org.springframework.integration.MessageHandlingException: error occurred in message handler [org.springframework.integration.x.redis.RedisChannelRegistry$CompositeHandler@26f8f92e] org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:76) org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:67) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:137) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:92) org.springframework.integration.x.redis.RedisQueueInboundChannelAdapter.access$3(RedisQueueInboundChannelAdapter.java:1) org.springframework.integration.x.redis.RedisQueueInboundChannelAdapter$ListenerTask.run(RedisQueueInboundChannelAdapter.java:145) org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:722) Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'jobTriggerBean'; nested exception is org.springframework.integration.MessageHandlingException: error occurred in message handler [org.springframework.integration.x.redis.RedisChannelRegistry$CompositeHandler@26f8f92e] org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:170) org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51) org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:339) org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:143) org.springframework.context.support.DefaultLifecycleProcessor.start(DefaultLifecycleProcessor.java:89) org.springframework.context.support.AbstractApplicationContext.start(AbstractApplicationContext.java:1278) org.springframework.xd.module.SimpleModule.start(SimpleModule.java:152) org.springframework.xd.dirt.module.ModuleDeployer.deployModule(ModuleDeployer.java:162) org.springframework.xd.dirt.module.ModuleDeployer.handleDeploy(ModuleDeployer.java:149) org.springframework.xd.dirt.module.ModuleDeployer.handleMessageInternal(ModuleDeployer.java:120) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:601) org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69) org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:97) org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:82) org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102) org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:103) org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:144) org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:231) org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:130) org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:73) ... 22 more Caused by: org.springframework.integration.MessageHandlingException: error occurred in message handler [org.springframework.integration.x.redis.RedisChannelRegistry$CompositeHandler@26f8f92e] org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:79) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.xd.dirt.plugins.job.JobTriggerBean.executeBatchJob(JobTriggerBean.java:74) org.springframework.xd.dirt.plugins.job.JobTriggerBean.start(JobTriggerBean.java:63) org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:167) ... 45 more Caused by: org.springframework.integration.x.json.TypedJsonMapper$SmartJsonConversionException: Infinite recursion (StackOverflowError) (through reference chain: org.springframework.batch.core.JobExecution["stepExecutions"]->java.util.UnmodifiableRandomAccessList[0]->org.springframework.batch.core.StepExecution["jobExecution"]->org.springframework.batch.core.JobExecution["stepExecutions"]->java.util.UnmodifiableRandomAccessList[0]->org.springframework.batch.core.StepExecution["jobExecution"]->org.springframework.batch.core.JobExecution["stepExecutions"]->java.util.UnmodifiableRandomAccessList[0]->org.springframework.batch.core.StepExecution["j ... ... >java.util.UnmodifiableRandomAccessList[0]->org.springframework.batch.core.StepExecution["jobExecution"]->org.springframework.batch.core.JobExecution["stepExecutions"]) com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:611) com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:492) com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:571) com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597) com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:492) com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:119) com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:23) com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serializeWithType(AsArraySerializerBase.java:197) com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:571) com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597) com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:492) | 2 |
| 693 | XD-780 | Avoid use of module name twice in location when using a custom modules | See https://github.com/SpringSource/spring-xd/pull/240#discussion_r6045724 | 2 |
| 694 | XD-781 | Support a dependencies-manifest aware Module Registry | It would be nice to be able to have modules that are simply made of: - Their context xml file - Some kind of manifest that expresses dependencies and have the runtime take care of the deps | 8 |
| 695 | XD-782 | Add CompositeModuleRegistry | NULL | 3 |
| 696 | XD-783 | Support higher level structure for complex module registry | See discussion at https://github.com/SpringSource/spring-xd/pull/240#discussion_r6045724 | 8 |
| 697 | XD-784 | Build needs to override $XD_* environment variables | export XD_HOME=foo gradle clean test build fails. Need to detected environment variables and override for the build | 2 |
| 698 | XD-785 | Enable profile selection from module options | This came up when working on email source. There is <int-mail:imap-idle-channel-adapter> and <int-mail:inbound-channel-adapter> It would be nice to be able to put those in two profiles and have one of the profile being activated from module options (e.g. email --polling=true|false) Don't know the runtime cost of activating profiles, but we could blindly activate profiles from all options passed explicitly : <beans profile="profile-[optionname]-[optionvalue]"> Not sure if this is the same as XD-132 | 5 |
| 699 | XD-786 | Add Warning-level log to postProcessAfterInitialization if Job name is not "job" | Should it be fatal vs. warning? | 1 |
| 700 | XD-787 | Add a JobExecution DTO Object | related to XD-779. * We need the ability to provide JSON serializable JobExecution information. * Change from using JavaSerialization back to returning objects | 2 |
| 701 | XD-788 | Add Integration Tests to run JobCommands Tests against all transports | similar to ChannelRegistry: - AbstractChannelRegistryTests that has the real tests - subclasses for each impl provide the registry to be tested Thus one test can run against multiple transports. | 8 |
| 702 | XD-789 | Add index-based access to TuplePropertyAccessor | NULL | 2 |
| 703 | XD-790 | Cryptic gradle error running tests when XD SingleNode is running | SingleNodeMain.launchSingleNodeServer(options) calls System.exit() causing a gradle buffer underflow. This is called from SingleNodeMainIntegrationTests. System.exit() should be called from the main method instead. | 1 |
| 704 | XD-791 | Document mail related sources & sinks | NULL | 5 |
| 705 | XD-792 | Update twittersearch to use Spring Integration support | NULL | 2 |
| 706 | XD-793 | Upgrade Spring Data Redis to 1.1 RC1 | NULL | 1 |
| 707 | XD-794 | Add integration tests for SpEL and Groovy based routing | NULL | 4 |
| 708 | XD-795 | Refactor mail and imap source into one "mail" module, leveraging Profiles | Once XD-785 is merged | 3 |
| 709 | XD-796 | Create separate commands for "--all" shell commands | Commands like "stream deploy" have changed over time to allow passing a "--all" option. So it's either {{stream deploy foo}} or {{stream deploy --all}}. This has a number of drawbacks, given that these are the only 2 alternatives: - Implementation code is cumbersome - None of the options can be marked mandatory, yet one of them is required. This has to be checked in the command code itself - TAB completion is less powerful as the shell doesn't know if we want the first or the second form. Consider splitting those commands into two distinct commands, one as before and one literally named {{stream deploy all}}. | 5 |
| 710 | XD-797 | Package Tangle Introduced by XD-790 | https://sonar.springsource.org/drilldown/measures/7173?metric=package_tangle_index&rids%5B%5D=7717 | 1 |
| 711 | XD-798 | Add support for ( ) grouping in Parser | NULL | 3 |
| 712 | XD-799 | Change rabbitmq sink to use routing-key-expression instead of routing-key | The current rabbitmq sink uses the attribute routing-key, which defaults to the name of the stream. This should be change to use the attribute routing-key-expression so that the routing-key can be determined using SpEL. This will enable a dynamic evaluation of the routing-key based on message payload/header. *Implementation Suggestions* This hopefully should be changing the XML description of the sink to routing-key-expression="${routingKey:'${xd.stream.name}'} This way, the ${xd.stream.name} is surrounded by a single quote to indicate a string literal to SpEL in the default case. *How to verify it works.* One of the simple uses of this is to create a routing key based on payload. In a distributed word-count example, the hashcode of a word would be sent to a certain number of processing modules that would perform the count. The idea is that the same word is sent to the same node over and over again, in particular if in-memory counters or state is computed - using centralized redis counters this wouldn't be necessary in the case of only counter state. The stream http | rabbit --routingKey="'word-' + payload.hashCode() % 3" is an example of a stream that can be used to verify that messages published to a direct exchange will have routing keys of the value word-0, word-1, and word-2. Binding a queue to each of these routing keys, one can observe the contents of messages in the queue to make sure that words are being routed to the appropriate queue, e.g. publishing "hello" as the payload of an http request should always appear in the same queue. The rabbitmq admin console can be used for this purpose. | 3 |
| 713 | XD-800 | Job channels need to denote a namespace | Job channels need to have a namespace. i.e. job-somejobname. Where the - is the delimiter for the namespace. The preference is to use the : instead of the -. But XD-766 needs to be completed in order to support this. | 2 |
| 714 | XD-801 | Turning on Regex Pattern multiline mode in JsonStringToObjectTransformer | Based on the PR discussion: https://github.com/SpringSource/spring-xd/pull/270#commitcomment-4003291 We need to consider turning on multiline mode for pattern matching. | 1 |
| 715 | XD-802 | Document splitter & aggregator processors | NULL | 3 |
| 716 | XD-803 | Batch Jobs need container & admin profiles | This is to set the appropriate data source, so that the container will use admins batch repository. | 3 |
| 717 | XD-804 | Add Named Channel API | We need an abstraction in place to retrieve messages from a "named channel" programmatically. Right now there is no implementation agnostic way of doing this (such as receiveMessage(), queueSize()). This could be quite useful for integration tests of streams. E.g. to do more focussed tests without resorting to "temp-files" and non-essential sinks or sources etc. - e.g. {code} :routeit > router --expression=payload.contains('a')?':foo':':bar' {code} | 8 |
| 718 | XD-805 | Get notified when created named channel "is ready" | For testing purposes it would be super-helpful if there be a hook to get notified when a named channel is up and running. In current tests one may have to resort to "Thread.sleep". | 8 |
| 719 | XD-806 | Validate module properties | Currently it's possible to do something like {code} http --prot=8888 {code} It is possible to validate property names by parsing the module definition file(s) directly and matching property placeholders (or profile declarations that may be mapped to properties, etc). This must account recursively for imports as well. (I have some code in a branch that does this). | 3 |
| 720 | XD-807 | Shell: Standardize counter name parameter | The parameters are not optimal for the counter name between "Aggregate Counter" "Field Value Counter" --counterName versus --name | 2 |
| 721 | XD-808 | Update to spring-data-hadoop 1.0.1.RELEASE | This might mean we should adjust our hadoopDistro options to the ones supported in the new release - hadoop12 (default), cdh4, hdp13, phd1 and hadoop21 | 3 |
| 722 | XD-809 | Shell integration with XD on Yarn | We should provide a better shell integration when XD is run on Yarn. 1. yarn kill --id TAB completion 2. yarn submit, more options like app name 3. yarn list, filter by app states, etc 4. admin config server TAB completion for running xd apps on yarn | 1 |
| 723 | XD-810 | Deleting a stream with reference to named channel disconnects channel from all streams | The following sequence results in "Dispatcher has no subscribers" error (stack trace below), because deleting stream2 disconnects stream1 from the foo channel. Current work on XD-685 has infrastructure for disconnecting just the channels involved in a stream, so should make it easier to fix this issue once merged. stream create stream1 --definition "time > :foo" stream create stream2 --definition "http > :foo" stream create stream3 --definition ":foo > file" stream destroy stream2 // expect file sink to still get time, but instead blows up b/c // deleteOutbound("foo") killed links b/w foo and both local output channels Server stack trace: 10:47:11,921 ERROR task-scheduler-6 handler.LoggingHandler:140 - org.springframework.integration.MessageDeliveryException: Dispatcher has no subscribers for channel 'SimpleModule [name=time, type=source, group=stream1, index=0].output'. org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:81) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:97) org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:199) org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:51) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:141) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:273) org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:268) org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:109) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ... 24 more 10:47:12,924 ERROR task-scheduler-6 handler.LoggingHandler:140 - org.springframework.integration.MessageDeliveryException: Dispatcher has no subscribers for channel 'SimpleModule [name=time, type=source, group=stream1, index=0].output'. org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:81) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:97) org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:199) org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:51) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:141) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:273) org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:268) org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:109) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ... 24 more 10:47:13,926 ERROR task-scheduler-4 handler.LoggingHandler:140 - org.springframework.integration.MessageDeliveryException: Dispatcher has no subscribers for channel 'SimpleModule [name=time, type=source, group=stream1, index=0].output'. org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:81) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:97) org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:199) org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:51) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:141) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:273) org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:268) org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:109) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ... 24 more 10:47:14,928 ERROR task-scheduler-4 handler.LoggingHandler:140 - org.springframework.integration.MessageDeliveryException: Dispatcher has no subscribers for channel 'SimpleModule [name=time, type=source, group=stream1, index=0].output'. org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:81) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:97) org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:199) org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:51) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:141) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:273) org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:268) org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:109) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ... 24 more 10:47:15,930 ERROR task-scheduler-1 handler.LoggingHandler:140 - org.springframework.integration.MessageDeliveryException: Dispatcher has no subscribers for channel 'SimpleModule [name=time, type=source, group=stream1, index=0].output'. org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:81) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.SourcePollingChannelAdapter.handleMessage(SourcePollingChannelAdapter.java:97) org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:199) org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:51) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:143) org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:141) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:273) org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49) org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:268) org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) | 3 |
| 724 | XD-811 | Add REST endpoint for launching Job | We need a REST endpoint to launch a job. Given the constraint on JobRegistry being not persistent and not available outside the container JVM, we can not use the batch job admin controller/service to launch the job. One possible way is to use the trigger source to launch the job at XD. | 2 |
| 725 | XD-812 | Re-enable support for tapping labels and named channels | As of XD-685, we no longer have the ability to: 1) Tap a named channel, ala stream1=:foo > sink stream2= :tap:foo > sink 2) Tap a stream whose source is a named channel ala stream2=:tap:stream1 > sink 3) Tap a label ala stream1=http | obfuscator: transform --expression=payload.replaceAll('password','*') | file stream2=:tap:stream1.obfuscator > sink Loss of named channel support is due to the fact that we are creating a WireTap on a module's local output channel only (thus we never tap named channels). We are not supporting labels because we only create a named channel called "tap:stream.module" on stream creation, so later creation of tap on stream1.obfuscator is referring to a non-existent named channel. | 8 |
| 726 | XD-813 | Destroying XD job should remove job's entries at batch job repositories/batch job locator | When an XD job is destroyed/deleted, the batch jobRepository entries for the job (associated JobInstances, JobExecutions etc.,) and the BatchJobLocator entries. | 4 |
| 727 | XD-814 | Rebase UI on top of new batch admin API | Now that the new batch admin api is taking shape, we need to rebase the XD web UI to use this. It's more than just changing the http urls sent to the xd server since the new API is not identical to the old one. | 5 |
| 728 | XD-815 | Updgrade to use spring-batch-admin 1.3.0 M1 when available | Once spring-batch-admin 1.3.0.M1 is available, update the build to use it. Likely to be Sept 7 or 9 | 1 |
| 729 | XD-816 | Support for composed streams | Some recent changes caused this to be turned off. Basically the change was to police whether a stream is well formed at create time, rather than deploy time. By deferring that check we can create composed streams that are not deployable by themselves but that are when used as building blocks in proper streams. | 2 |
| 730 | XD-817 | REST API for job listing should provide details on last execution | A user should be able to view some important details of the last execution of a job from a job list. The {{/batch/jobs}} REST endpoint should provide extra fields not currently available in the {{JobInfo}} class. At a minimum, I would like to see: * startTime * startDate * last job parameters * duration * last job status | 5 |
| 731 | XD-818 | UI should poll server for latest on job info | The admin UI should be polling the server to automatically pick up any new jobs, executions, and instances. | 5 |
| 732 | XD-819 | Add Service Activator Processor | Would be nice to have a ServiceActivator Processor available so that if one had an existing Spring bean they could simply describe the bean id and method name - without going through the full complexity of creating a processing module. | 3 |
| 733 | XD-820 | Jobs and taps should not require a leading : since they have name spaces. | NULL | 3 |
| 734 | XD-821 | Change url to access UI from browser | Currently, the url for accessing the XD UI is {{http://localhost:8080/admin-ui/index.html}}. This feels messy and dated. We should be able to access the ui without explicitly including the index.html, like this: {code} http://localhost:8080/admin-ui {code} | 5 |
| 735 | XD-822 | TupleBuilder.fromString() should not overwrite original id and timestamp fields | When converting a JSON string to a tuple the JSON may contain id. This method should handle this. Same with timestamp | 2 |
| 736 | XD-823 | add discardDeletes property to twitterstream source | If true(default): filter for delete messages in the twitter stream and route to a discard channel. This creates a twitter stream including only new tweets and no references to deleted ones. | 1 |
| 737 | XD-824 | Rename ChannelRegistry | My current thinking is... ChannelRegistry -> MessageBus RabbitChannelRegistry -> RabbitMessageBus ... Then, method names change like so: createInbound -> registerConsumer createOutbound -> registerProducer | 3 |
| 738 | XD-825 | Create object-to-json transformer processor | <int:channel id="input"/> <int:object-to-json-transformer input-channel="input" output-channel="output"/> <int:channel id="output"/> | 1 |
| 739 | XD-826 | Investigate failing tests in MailCommandTest | org.springframework.xd.shell.command.MailCommandTests in spring-xd-shell project. I get failures relating to invalid username/password INFO: Stream Name Stream Definition Status ----------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------- mailstream imap --port=1044 --protocol=imap --folder=INBOX --username=johndoe --password=secret | file --dir=/tmp --name=FileSink1280066074228960855 --suffix=txt --charset=UTF-8 --binary=false deployed 13/09/04 11:32:11 WARN mail.ImapIdleChannelAdapter: error occurred in idle task javax.mail.AuthenticationFailedException: LOGIN failed. Invalid login/password at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:663) | 3 |
| 740 | XD-827 | provide a property on twittersearch to enable the object-to-json transformer | Twitter search source should produce JSON or Pojo. The Pojo requires a custom wrapper class that is JSON friendly (e.g., zero arg constructor). The twittersearch module should have a parameter --json true/false (default true) to control the output type. | 2 |
| 741 | XD-828 | rename Bridge to Binding and add direction | org.springframework.integration.x.bus.Bridge should now be called Binding we can also move the INBOUND/OUTBOUND direction (or possibly the CONSUMER/PRODUCER role) into this class; that should simplify its usage in conditional code within the MessageBus code and also reduce the use of "in" and "out" as Strings in that same code | 2 |
| 742 | XD-829 | Upgrade hsqldb version on XD batch admin to the latest | We also would like to upgrade the hsqldb version on spring batch admin so that both are compatible. | 3 |
| 743 | XD-830 | Expose Job parameters for JobExecutions | Currently, the way that we are accessing JobExecutions from batch admin, the JobParameters are not being filled in. We are using a {{org.springframework.batch.admin.service.JdbcSearchableJobExecutionDao}} to grab the job executions from the database. The database queries that it uses does not include the JobParameters (which is stored in a different table). I think that this will require a change to batch admin in order to properly expose the parameters. | 5 |
| 744 | XD-831 | Update docs to cover Module config and lib directory structure | NULL | 2 |
| 745 | XD-832 | Update docs for new Tap syntax | Now that taps are just channels, we need to update the docs. The preceding colon is no longer needed (and will be removed altogether), so all examples should be like this: {code} tap:foo > bar {code} | 2 |
| 746 | XD-833 | Allow tapping a stream prior to stream creation without specifying module name | Currently this works: stream create foo --definition "tap:baz.time > log" stream create baz --definition "time | file" But this doesn't: stream create foo --definition "tap:baz > log" stream create baz --definition "time | file" This is because the parser translates references to "tap:baz" to named channel "tap:baz.time" (the name of the stream's first module). If the stream is not yet created, the parser cannot perform this translation. A fix for this will likely be related to the fix needed for XD-812. | 5 |
| 747 | XD-834 | Handle XD admin server shutdown cleanly | There are couple of issues here: 1) The admin server destroy() - close event's onApplicationEvent(ContextClosedEvent) listener has stop() to stop the admin server's tomcat instance. The stop() also calls the applicationContext's destroy() which loops again to stop. 2) With HSQLServer or any batch db server(in future), the admin server stop() also needs to handle the batch db server shutdown. | 2 |
| 748 | XD-835 | REST - Listing of FieldValueCounter not working | http://localhost:8080/metrics/field-value-counters {code} <errors xmlns:atom="http://www.w3.org/2005/Atom"> <error logref="HttpMessageNotWritableException"> <message> Could not marshal [PagedResource { content: [links: [<http://localhost:8080/metrics/field-value-counters/hashtags>;rel="self"]], metadata: Metadata { number: 0, total pages: 1, total elements: 1, size: 0 }, links: [] }]: null; nested exception is javax.xml.bind.MarshalException - with linked exception: [com.sun.istack.SAXException2: unable to marshal type "org.springframework.xd.rest.client.domain.metrics.MetricResource" as an element because it is not known to this context.] </message> </error> </errors> {code} | 4 |
| 749 | XD-837 | Tipsy tooltip hovers are not responsive after jobs list is refreshed from server | In the XD UI, the list of jobs is refreshed from the server every 5 seconds. There are also tooltips that are activated when hovering over a job execution. These tooltips are no longer responsive (ie- they remain on the page and don't disappear) after the list of jobs is refreshed. | 5 |
| 750 | XD-838 | Refactor out Trigger docs from the Batch Job chapter | NULL | 2 |
| 751 | XD-839 | Close parent contexts when shutting down | NULL | 1 |
| 752 | XD-840 | Fix Compiler Warnings | As of M3... {code} javadoc: warning - Error fetching URL: http://static.springsource.org/spring-shell/docs/current/api/package-list /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/module/CompositeModuleRegistry.java:40: warning - @param argument "cp" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/module/CompositeModuleRegistry.java:40: warning - @param argument "file" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/plugins/job/JobFactoryBean.java:62: warning - Tag @link: can't find jobName in org.springframework.xd.dirt.plugins.job.JobFactoryBean /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/rest/XDController.java:56: warning - @param argument "<V>" is not a type parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/rest/XDController.java:56: warning - @param argument "<T>" is not a type parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/plugins/job/batch/BatchJobAlreadyExistsException.java:32: warning - @param argument "message" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/plugins/job/batch/BatchJobLocator.java:32: warning - @DistributedJobService is an unknown tag. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/plugins/job/batch/BatchJobLocator.java:32: warning - @DistributedJobService is an unknown tag. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/plugins/job/batch/NoSuchBatchJobException.java:31: warning - @param argument "message" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/server/options/ContainerOptions.java:30: warning - @param argument "defaultTransport" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/server/options/ContainerOptions.java:30: warning - @param argument "defaultAnalytics" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/server/options/OptionUtils.java:35: warning - @return tag has no arguments. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-module/src/main/java/org/springframework/xd/module/Plugin.java:32: warning - Tag @see: reference not found: ModuleDeployer /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/bus/MessageBus.java:53: warning - @param argument "moduleInputChannel" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/bus/MessageBus.java:72: warning - @param argument "moduleOutputChannel" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/bus/MessageBus.java:102: warning - @param argument "moduleOutputChannel" is not a parameter name. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-reactor/src/main/java/org/springframework/xd/integration/reactor/config/ReactorNamespaceUtils.java:46: warning - @return tag has no arguments. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/plugins/job/batch/BatchJobLocator.java:32: warning - @DistributedJobService is an unknown tag. /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/plugins/job/batch/BatchJobLocator.java:32: warning - @DistributedJobService is an unknown tag. {code} and {code} warning: [options] bootstrap class path not set in conjunction with -source 1.6 /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/bus/LocalMessageBus.java:132: warning: [rawtypes] found raw type: LocalMessageBus.SharedChannelProvider SharedChannelProvider channelProvider = aliasHint ? queueChannelProvider ^ missing type arguments for generic class LocalMessageBus.SharedChannelProvider<T> where T is a type-variable: T extends AbstractMessageChannel declared in class LocalMessageBus.SharedChannelProvider /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/bus/LocalMessageBus.java:159: warning: [rawtypes] found raw type: LocalMessageBus.SharedChannelProvider SharedChannelProvider channelProvider = aliasHint ? queueChannelProvider ^ missing type arguments for generic class LocalMessageBus.SharedChannelProvider<T> where T is a type-variable: T extends AbstractMessageChannel declared in class LocalMessageBus.SharedChannelProvider /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDEntities.java:93: warning: [rawtypes] found raw type: List public List getUrls() { ^ missing type arguments for generic class List<E> where E is a type-variable: E extends Object declared in interface List /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDEntities.java:101: warning: [rawtypes] found raw type: List public List getHashTags() { ^ missing type arguments for generic class List<E> where E is a type-variable: E extends Object declared in interface List /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDEntities.java:109: warning: [rawtypes] found raw type: List public List getMentions() { ^ missing type arguments for generic class List<E> where E is a type-variable: E extends Object declared in interface List /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDEntities.java:117: warning: [rawtypes] found raw type: List public List getMedia() { ^ missing type arguments for generic class List<E> where E is a type-variable: E extends Object declared in interface List /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDEntities.java:126: warning: [rawtypes] found raw type: List public List getTickerSymbols() { ^ missing type arguments for generic class List<E> where E is a type-variable: E extends Object declared in interface List /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDEntities.java:37: warning: [serial] serializable class XDEntities has no definition of serialVersionUID public class XDEntities implements Serializable { ^ /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDUrlEntity.java:31: warning: [serial] serializable class XDUrlEntity has no definition of serialVersionUID public class XDUrlEntity implements Serializable { ^ /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDHashTagEntity.java:32: warning: [serial] serializable class XDHashTagEntity has no definition of serialVersionUID public class XDHashTagEntity implements Serializable { ^ /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDMentionEntity.java:32: warning: [serial] serializable class XDMentionEntity has no definition of serialVersionUID public class XDMentionEntity implements Serializable { ^ /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDMediaEntity.java:32: warning: [serial] serializable class XDMediaEntity has no definition of serialVersionUID public class XDMediaEntity implements Serializable { ^ /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDTickerSymbolEntity.java:31: warning: [serial] serializable class XDTickerSymbolEntity has no definition of serialVersionUID public class XDTickerSymbolEntity implements Serializable { ^ /Users/gpr/Documents/workspace-si/spring-xd/spring-xd-dirt/src/main/java/org/springframework/integration/x/twitter/XDTweet.java:34: warning: [serial] serializable class XDTweet has no definition of serialVersionUID public class XDTweet implements Serializable { ^ {code} | 1 |
| 753 | XD-841 | Create a packaging model for custom modules | Like just discussed after the Spring One session, it would be nice to have a packaging model for custom modules. Instead of putting all libs in one directory and an XML in another, it could be a fat jar including some metadata file pointing to either an XML or to a JavaConfig configuration class inside the jar. Some more words why I think it's important to have something like that: Spring XD can be very well used in the enterprise, and at least for job modules it's likely to have some dependencies. So what do you have to do now to deploy such a module? Build a jar with your build process, then get it from your repository, unpack it and somehow find the relevant XML to copy it into the appropriate folder, then inspect the POM to find out what all the dependencies are, direct and transient, and get them from your repository to copy them into the appropriate folder. And of course, copy the jar. And when using Spring XD in distributed mode, you have to do it several times. With the approach mentioned above you just build your fat jar with your build process and distribute it to all your nodes, nothing complicated. And if you implement it like that it's easy to have a JavaConfig support that's almost the same. | 9 |
| 754 | XD-842 | Add back classifier = 'dist' to distZip build target | Add back "classifier = 'dist'" to distZip build target - it was was accidentally removed. | 1 |
| 755 | XD-843 | Initial XD on CloudFoundry support | First take on this involves - being able to deploy the two separate applications: xd-admin & xd-container - being able to CF-service provided redis & rabbit for internal needs of XD - to some extent, make modules smart and CF aware (e.g. http source uses correct port) | 20 |
| 756 | XD-844 | Enable Spring Boot Loader support | As part of running in Cloud Foundry, one quick workaround for the lack of "classpath" support would be to use the Spring Boot Loader special ClassLoader and jar-inside-a-jar support: https://github.com/spring-projects/spring-boot/tree/master/spring-boot-tools/spring-boot-loader | 3 |
| 757 | XD-845 | Remove support for the leading : on items that have a declared namespace | When using jobs, taps we no longer need to have the leading :. i.e. :tap:foo. We should only support tap:foo. | 3 |
| 758 | XD-846 | Remove deprecated tap syntax from the parser. | Tap '@' and using numbers instead of module names. | 2 |
| 759 | XD-847 | Revise the available hadoopDistro options | We should adjust our --hadoopDistro options to the ones supported in the new spring-data-hadoop 1.0.1.RELEASE - hadoop12 (default), cdh4, hdp13, phd1, hadoop20 This includes updating the wiki pages | 5 |
| 760 | XD-848 | ICLA website is missing Spring XD project option | The README.md at: https://github.com/spring-projects/spring-xd/blob/master/README.md ...says: "Before we accept a non-trivial patch or pull request we will need you to sign the contributor's agreement" Navigating to the <contributor's agreement> link takes the user to the ICLA web page at: https://support.springsource.com/spring_committer_signup //exp: The Project field's dropdown should have an option for "Spring XD" //act: There's no "Spring XD" option | 1 |
| 761 | XD-849 | Gemfire modules should support connection via locator | The gemfire modules currently accept server host and port. Provide an option to specify a locator host and port | 2 |
| 762 | XD-850 | JAR version mismatches | Looks like there are some version mismatch issues with the build/packaging of the XD components. Looking in xd/lib I see the following which looks suspicious: mqtt-client-0.2.1.jar mqtt-client-1.0.jar jackson-core-asl-1.9.13.jar jackson-mapper-asl-1.9.12.jar spring-integration-core-3.0.0.M3.jar spring-integration-http-2.2.5.RELEASE.jar spring-data-commons-1.6.0.M1.jar spring-data-commons-core-1.4.0.RELEASE.jar | 3 |
| 763 | XD-851 | Refactor DSL for Taps and Jobs Usage | The syntax for both taps and job channels will be prefixed with the word tap. * The parser will search both the all (stream and job for now) registries to find the "name". * If the name is present in only in one registry then that definition will be used. * If the name is present both registries a syntax exception will be issued. - The user then can optionally specify which registry to use by adding a second token to the definition that specifies the type of channel the user wants. - i.e. tap:{stream}:streamName > mySink * If the user attempts to place a job on the greater (left hand) side without specifying a notification a syntax error will be thrown. Format: -- tap:[type]:name > $ -- $ > tap:[type]:name -- Example: tap:streamName >mySink tap:stream:streamName > mySink mySource|myProcessor > tap:stream:mySink myEmailSource > tap:job:jobName.step1 myEmailSource > tap:jobName.step1 tap:job:jobName-notifications > myEmailSink | 4 |
| 764 | XD-852 | DSL needs to have wildcard support for taps | Wildcard support for associating inbound and outbound channels with modules. The wild card will be represented by an asterisk '*'. Example: myEmailSource > tap:job:* send message to all jobs myEmailSource > tap:* send message to all stream/job taps myEmailSource > :*foo* send message to all channels that contain the channels that contains the word 'foo' tap:*bar > myEmailSource | 5 |
| 765 | XD-853 | Add definition of serialVersionUID to Twitter classes | Add serialVersionUID to the objects in package org.springframework.integration.x.twitter: * XDEntities * XDUrlEntity * XDHashTagEntity * XDMentionEntity * XDMediaEntity * XDTickerSymbolEntity * XDTweet The absence creates warnings during compile time. | 2 |
| 766 | XD-854 | Update doc about modules and spring | The doc at http://docs.spring.io/spring-xd/docs/1.0.0.M3/reference/html/#_modules_and_spring refers to an old version of the counter sink, when it was still hardwired to use redis. The text next to it that explains placeholders is out of date (with respect to the redis placeholders) | 2 |
| 767 | XD-855 | Change metrics assertions in integration tests to use smart Thread.sleep | Similar to what has been done for e.g. FileSink, refactor metrics related sinks to use smart Thread.sleep() timings | 5 |
| 768 | XD-857 | Refactor FileModuleRegistry as "ResourceModuleRegistry" | Apart from sanity checks, there is not much that ties FileModuleRegistry to actual java.io.Files. Using the Resource abstraction would work just the same, and would allow loading modules from the classpath in constrained environments or other file systems/locations. (HDFS /HTTP) | 5 |
| 769 | XD-858 | Upgrade to Spring Data Redis 1.1.0.RELEASE | NULL | 1 |
| 770 | XD-859 | Support for listing of modules in the REST API | From the CLI, one should be able to get a listing of modules and be able to specifically ask for jobs, sources, sinks, and processors. A brief description of them would also be nice - this might come from adding some metadata into the definition. Finer grained description/implementation suggestion TBD | 8 |
| 771 | XD-861 | Type Conversion across modules | This story will need to be broken down further. The current code mixes together the type conversion that happens within a single JVM (for data that is passed on a local transport between modules) and serialization/deserialization between JVMs. This should be separated. There was a suggestion that we could perhaps use typed data channels in SI as a means implement the type conversion between modules. The media-type conversion support in Spring 4 is another part of this solution. | 10 |
| 772 | XD-862 | First class JSON Path support | Similar to xpath with XML, there are now some initial support in SI that enable the use of filter/routers based on JSON Path expressions. That support needs to be reviewed and then brought up to the level of exposure in Spring XD so that router/filter modules could use JSON Path. json-path-filter/router are components that need to be created, perhaps others. This story needs to be broken down further. | 10 |
| 773 | XD-863 | Serialization of Spring Batch Context Objects | Needs some investigation on how the update information from Spring Batch listeners can be sent as a message across modules in a single node configuration as well as across JVMs in a distributed node configuration. | 4 |
| 774 | XD-864 | Move BatchJobExecutionsByJobName to BatchJobsController from BatchJobExecutionsController | We need to move the BatchJobExecutionsByJobName method to BatchJobsController as that seems appropriate | 1 |
| 775 | XD-865 | Add a test suite to the admin-ui | The admin-ui currently has no unit tests. Need to add a test suite and hook it up to the build so that tests are run on every build. | 5 |
| 776 | XD-866 | Remove remaining Thread.sleeps from the job tests | Get rid of all the thread.sleeps and code that supported them. | 1 |
| 777 | XD-867 | Support the ability for a user to create composite modules that can accept parameters | Parser creates a module compose command that allows a user to create a module of other modules. This composed module can accept parameters. | 5 |
| 778 | XD-868 | Create microbenchmark performance test of reactor syslog adapter vs standard syslog adapter | We need to verify that we are seeing improved throughput when using the reactor based syslog adapter. A suggestion on a basic stream to perform a microbenchmark this would be using in-memory counters, singlenode with the stream definition "syslog | counter". Based on the results of this microbenchmark, other stories may need to be created. | 4 |
| 779 | XD-869 | Colocate Modules using labels. | i.e. http | group1: filter | group1: transform | file then specifying anything labelled group1 goes to machineX | 6 |
| 780 | XD-870 | Support for listing of modules in the Shell | Commands that pair up with the functionality described in XD-859 module list (would list all modules in a table format) module list --type=source (would list only source modules) and so on. | 4 |
| 781 | XD-871 | No errors in Shell when creating stream with HTTP Source + already used port | 2 issues exist: 1) Current this does not create an error in the shell {code} stream create --name s1 --definition "http | log" stream create --name s2 --definition "http | log" {code} On the server-side I see: {code} Caused by: java.net.BindException: Address already in use sun.nio.ch.Net.bind0(Native Method) sun.nio.ch.Net.bind(Net.java:344) sun.nio.ch.Net.bind(Net.java:336) sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193) org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290) org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) ... 3 more {code} 2) The Stream should not be saved to the StreamDefinitionRepo in case of an error. | 4 |
| 782 | XD-872 | Make in-memory meta data stores persistent | Just wanted to create story for this - so we can consider whether this should be addressed. In at least 2 modules we use non-persisted states. We may want to consider making them persistent: *Twitter Search* uses an in-memory *MetadataStore* that keeps track of the twitter ids. There exists a corresponding issue for Spring Integration: "Create a Redis-backed MetadataStore" See: https://jira.springsource.org/browse/INT-3085 *File Soure*'s File Inbound Channel Adapter uses a AcceptOnceFileListFilter, which uses an in-memory Queue to keep track of duplicate files. | 8 |
| 783 | XD-873 | File Source - Provide option to pass on File object | This story may need to be broken into several stories Particularly for Batch scenarios one may not want to run a "file-to-string-transformer" on the payload file in the file source but rather handle/pass the file reference itself (local SAN etc.) - e.g. in case somebody drops a 2GB or in scenarios where one wants to push those large files into HDFS and run hadoop jobs on the data. This is important for Batch Jobs as they need to access the file itself for the reader. We need to *keep in mind the various transports we support*. Not sure how Kryo handles file serialization. I would think we only need the File Meta Data to be persisted not the file-data itself (make that configurable??). | 8 |
| 784 | XD-874 | For file based item reader jobs, step/job completion message should have name of file sent on named channel | It looks like we don't handle deletion of source files currently. We should provide some support for that - Maybe there is a way to into Spring Integration's PseudoTransactionManager support: http://docs.spring.io/spring-integration/api/org/springframework/integration/transaction/PseudoTransactionManager.html The *File Source* should possibly also support File archival functionality (But that might also be a dedicated processor?). Not sure where we want to set the semantic boundaries for the File Source. | 8 |
| 785 | XD-875 | Handle AdminServer shutdown cleanly | The admin Server's tomcat is not shutdown properly. There is an existing method shutdownCleanly() on AdminServer but the spring-xd-shell tests hang when we use this method to shutdown the admin server. | 2 |
| 786 | XD-876 | Create FsShell based module to copy a File to HDFS | Spring batch workflows are great for complex hadoop operations but if I want to create a simple processor that executes some hadoop fs in groovy, it would be nice to do this: {code} <service-activator output-channel="output" input-channel="input"> <hadoop:script id="loadScript" language="groovy"> def outputPath = "${hdfsPath}" fsh.put("-", outputPath) </hadoop:script> </service-activator> {code} The goal of this hadoop script is to use in a stream like this:"file | script" that puts a file byte for byte to hdfs. An enhanced hdfs sink that's optimized for binary data like images/pdfs might be more elegant but I was hoping that this would work. This script gets ignored by Spring XD. But even if it didn't, I am not sure the "-" stdin put would work as hoped. | 1 |
| 787 | XD-877 | Taps do not work when JMX is enabled | When JMX is enabled, the Module's output channel processed by StreamPlugin is a Proxy. Thus it fails the "instanceof" test used to apply the WireTap to the output channel. | 3 |
| 788 | XD-878 | Remove code in StreamPlugin to extract target output channel from proxy | The SI JMX MBeanExporter proxies the output channel created in a module context when JMX is enabled. We have to unwrap the proxy in StreamPlugin in order to add the tap. XD-877 temporarily introduced this code, but a more elegant solution is called for. Perhaps this will involve SI making addInterceptor an interface method. | 3 |
| 789 | XD-879 | Cleanup hsqldb data directory used by tests after each test completion | Currently, the "data" directory created by the HSQLDB process during the tests run is not cleaned up and may cause issues. We should delete the "data" directory after each test completion. | 2 |
| 790 | XD-880 | Update Java Version to 7 | NULL | 1 |
| 791 | XD-881 | Introduce wire.js into the XD admin UI | We should consider moving to wire.js to encourage dependency injection in the UI Javascript code. See here: https://github.com/cujojs/wire/blob/master/docs/get.md | 5 |
| 792 | XD-882 | Disable the JMX setting in SingleNodeMainIntegrationTests.testConfiguration | Set the enableJmx to false because contexts are not getting destroyed properly, and in some cases prevents testSystemPropertiesOverridesDefault from running successfully. | 1 |
| 793 | XD-883 | Refactor the tests so that contexts are destroyed properly | In our tests the context is destroyed at the end of each module. It should be destroyed at the close of the container. | 5 |
| 794 | XD-884 | Do not initialize spring batch schema on each test run | If the spring batch database has already been initialized do not re-initialize for each test run. | 3 |
| 795 | XD-885 | Add Batch Job Listeners Automatically | Add Batch Job Listeners Automatically * Each major listener category should send notifications to own channel (StepExecution, Chunk, Item etc.) * Add attribute to disallow automatic adding of listeners | 8 |
| 796 | XD-886 | Fix package tangle between org.springframework.xd.dirt.plugins.job and org.springframework.xd.dirt.job | NULL | 1 |
| 797 | XD-887 | Package modules with their support jars | Currently, a lot of jars that are on the classpath of xd-dirt are there to support modules. We should move those to the lib/ construct of a module and remove them from the CP of dirt. But this should not be done by simple "mv", as we'd lose version tracking and dependency management offered by gradle. Pending a "dependency-aware" ModuleRegistry, we should be able to alter the build.gradle file so that it - knows about individual modules (maybe handles them as project) - copies its libs into the appropriate directory - does not copy dependencies that are already legitimate dependencies of xd-dirt (this can be achieved by runtime introspection of the dependeny tree of both projects) | 6 |
| 798 | XD-888 | Remove org. in hsqldb dependency | NULL | 1 |
| 799 | XD-889 | Change default admin port from 8080 | This conflicts with and 'out of the box' hadoop installation that uses 8080 as the 'map reduce shuffle port'. 8088 sound ok? | 1 |
| 800 | XD-890 | Run JavaScript tests (Jasmine) as part of the build process | We probably need to look into some options to run our JavaScript tests (Jasmine) as part of the build process - some possibilities: * Jasmine Gradle Plugin https://github.com/dzhaughnroth/jasmine-gradle-plugin * Saga - http://timurstrekalov.github.io/saga/ Looks like Maven has slightly better support: http://searls.github.io/jasmine-maven-plugin/index.html See also: XD-865 | 8 |
| 801 | XD-891 | Provide a way to access currently deployed modules | For testing, it would be useful to access the deployed Module instances to connect sources and or sinks to a module's input and output channels, etc. This could be a simple as exposing the deployedModuleMap on ModuleDeployer or possibly something more elaborate if this level of granularity is generally useful for runtime administration. | 3 |
| 802 | XD-892 | Spring Batch Behavior change from M2 to M3 | In M3, the batch job behavior has changed. In M2, it was much easier to create an invoke a batch job. In M3, a trigger is required. Figuring that change out isn't a big deal but the behavior of this batch job in M3 throws a stack trace, yet it executes. In M2, this same batch job runs fine with no stack trace. Logs are attached. I can't see a difference in the container log property files from M2 to M3. Turning the log settings down will suppress the traces, but I was not expecting the traces since they did not show up in M2. Stream Definitions: job create --name pdfLoadBatchJob --definition "batch-pdfload --inputPath='LOCAL_PDF_PATH' --hdfsPath='REMOTE_HDFS_PATH'" stream create --name pdfloadtrigger --definition "trigger > job:pdfLoadBatchJob" | 1 |
| 803 | XD-893 | XD UI: Job Parameters tool tips display needs to aligned | When clicking on a specific job execution from the job executions bar chart, the tool tips display isn't aligned with the job parameters. Please see the attachment. | 2 |
| 804 | XD-894 | Create an easier short-cut for launching adhoc Batch Jobs | Currently, for adhoc launching of Batch jobs you have to use: {code} stream create --name myTriggerStream --definition "trigger > job:helloSpringXD" {code} For renewed triggering of the job you have to undeploy and then redeploy the job. It would be nice if there was possibly a slightly simpler way of doing this. Just FYI - As a different approach you can also use the HTTP source: {source} job create --name myjob --definition "myjob" stream create --name myjobhttp --definition "http > job:http" http post --data "{}" {source} | 8 |
| 805 | XD-896 | Add --inputType and --outputType module parameters | The ability to configure message conversion via parameters. Consider programatic configuration of data type channels. Values can be media type, e.g., application/json or a java class name. | 8 |
| 806 | XD-897 | The HDFS Sink should support copying File payloads | We should support *java.io.file* payloads in order to support non-textual file and large text file payloads being uploaded to HDFS. Currently text file payloads are converted to a text stream in memory and, non-String payloads are converted to JSON first, using an "object-to-json-transformer". Ultimately we need to support streams such as "file | hdfs" where the actually payload being copied to HDFS is not necessarily JSON or textual. Need to be able to support headers in the message that will indicate which HDFS file the data should be stored in. | 8 |
| 807 | XD-898 | Create Integration Tests for Batch Notifications | NULL | 5 |
| 808 | XD-899 | xd-container should start even if xd-admin is not running | currently xd-container will not start due to a DB connection failure if the xd-admin is not already running In fact, if someone is not using Batch jobs at all with XD, they should not even need a DB connection for either xd-admin or xd-container to run so... consider using LazyConnectionDataSourceProxy so a connection failure would only occur when the DataSource is actually invoked to retrieve a connection | 2 |
| 809 | XD-900 | Move SpEL PropertyAccessors to Module Parent Context | When INT-3133 is resolved, SpEL {{PropertyAccessor}} s are inherited from parent contexts. Instead of adding the {{JsonPropertyAccessor}} to each module's context, add it to the parent instead. | 1 |
| 810 | XD-901 | Wrong Jetty Util on classpath for WebHdfs | We currently include jetty-util-6.1.26.jar but we need to add correct jar for different distributions - PHD uses jetty-util-7.6.10.v20130312.jar Need to check hadoop-hdfs dependencies for the distros and add jetty-util-* to the jar copy for each distro | 3 |
| 811 | XD-902 | Properly close Redis/Rabbit connection factories in tests | Tests that leverage [Redis|Rabbit]AvailableRule often create another connection factory in the test body but fail to close it. Tests should properly close the resource. As an added benefit, the rule itself can expose the resource that it created for deciding whether to skip the test or not | 2 |
| 812 | XD-903 | Split xd-dirt in 3 (or 5) | The xd-dirt project should be split in at least 3 parts: - Classes and resources pertaining to the admin-server - Container server - Shared classes Additionally, we may consider splitting the first two in half as well, having a separate project for CLI handling (and hence introduce 2 other projects for YARN, etc) | 8 |
| 813 | XD-904 | Fix hardcoded redis port from tests | kparikh-mbpro:spring-xd kparikh$ grep -r 6379 * | grep java spring-xd-analytics/src/test/java/org/springframework/xd/analytics/metrics/common/RedisRepositoriesConfig.java: cf.setPort(6379); spring-xd-analytics/src/test/java/org/springframework/xd/analytics/metrics/integration/GaugeHandlerTests.java: cf.setPort(6379); spring-xd-analytics/src/test/java/org/springframework/xd/analytics/metrics/integration/RichGaugeHandlerTests.java: cf.setPort(6379); spring-xd-dirt/src/test/java/org/springframework/xd/dirt/listener/RedisContainerEventListenerTest.java: cf.setPort(6379); | 1 |
| 814 | XD-905 | Container start/stop publish events are not getting processed | It looks like Container's ContainerStartedEvent and ContainerStoppedEvent are published from ContainerLauncher's context whereas the ContainerEventListeners are running in XDContainer's context. This makes the container start/stop events not getting processed. | 2 |
| 815 | XD-906 | Add aggregate counter monthly resolution query support | NULL | 5 |
| 816 | XD-907 | Add aggregate counter year resolution query support | NULL | 3 |
| 817 | XD-908 | Add aggregate counter query by number of points | It should be possible to supply a start or end date (or none for the present), plus a "count" value for the number of points required (i.e. after or prior to the given time). | 3 |
| 818 | XD-909 | Support additional aggregate counter query options | NULL | 5 |
| 819 | XD-910 | Add a Processor for Restful webservices | Offers the functionality to make http request to a web service. i.e. outbound http gateway. Example implementations: stream create --name foo --definition "trigger |rest --reply-timeout=1 --url=http://earthquake.usgs.gov/earthquakes/feed/geojson/all/day|log" stream create --name foos --definition "trigger --payload=lat=34.0567006&lon=-84.34368810000001&site=all&smap=1&searchresult=Roswell%2C%20GA%2030076%2C%20USA#.UktzaWSG1Dd | rest --url=http://forecast.weather.gov/MapClick.php? |log" | 1 |
| 820 | XD-911 | Reduce Sonar Critical Errors | Should keep the critical error count as close to zero as possible. | 1 |
| 821 | XD-912 | Support for registering custom message converters | Users need to register custom message converters used by modules. | 5 |
| 822 | XD-913 | The XD build breaks with Gradle 1.8 | The XD build breaks with Gradle 1.8 due to some changes in dependency resolution. | 1 |
| 823 | XD-914 | Add documentation for #jsonPath functionality with SpEL based processors | See issue https://jira.springsource.org/browse/XD-862 The docs should be updated to include examples that show how to use the standard 'SpEL' based splitter, transformer, filters with #jsonPath expressions. | 2 |
| 824 | XD-915 | Convert modules to be CP-aware | Once XD-887 is merged, gradually convert more modules. Recipe: 1) Move the <module>.xml file to <module>/config/<module>.xml 2) Declare a :module.<type>.<module> gradle project 3) Move dependencies from dirt project to newly created module project 4) gradle build picks it up. gradle clean build + manual test Also have a look at gradle cleanEclipse eclipse | 8 |
| 825 | XD-916 | File source should be able to produce file contents or file reference | File source should output either the File itself (serialized File object) or the contents as a byte[]. This option is configured by a parameter --contents=true. The byte[] may be converted to a String using XD Message Conversion, e.g., --output = text/plain;charset=UTF-8 | 4 |
| 826 | XD-917 | Make the parser aware of message conversion configuration | Enhance the stream parser to take message conversion into account in order to validate or automatically configure converters. For example: {noformat:nopanel=true} source --outputType=my.Foo | sink --inputType=some.other.Bar is likely invalid since XD doesn't know how to convert Foo->Bar. {noformat} | 8 |
| 827 | XD-918 | Refine project (mainly dirt) dependencies | See also XD-903, XD-915 A lot of dependencies have been added with the "compile" scope as an oversight over time. Some of them are only required at runtime, some may not be required anymore. | 1 |
| 828 | XD-919 | Remove json parameter from twittersearch source | json parameter is no longer required. Use --outputType=application/json instead | 2 |
| 829 | XD-920 | MQTT source module does not cleanly undeploy | Was attempting to test mqtt, turns out I don't have the proper rabbitmq thing installed. So far so good, I get these kinds of exceptions: {noformat} Unable to connect to server (32103) - java.net.ConnectException: Connection refused org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:75) org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:521) java.lang.Thread.run(Thread.java:724) Caused by: java.net.ConnectException: Connection refused java.net.PlainSocketImpl.socketConnect(Native Method) java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) java.net.Socket.connect(Socket.java:579) org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:66) ... 2 more {noformat} Problem is, I still get them after undeploying my "mqtt | log" stream | 2 |
| 830 | XD-921 | Add more "hands on" example to MQTT doco | Not everyone may be familiar with MQTT, or esp. with MQTT inside Rabbit | 3 |
| 831 | XD-922 | Handle SingleNodeServer's stop() method cleanly | SingleNode server needs to stop cleanly with stopping both the admin server & container server. Also, all the tests that require SingleNode main server needs to handle the server shutdown appropriately. | 2 |
| 832 | XD-923 | Error Channel for streams modules that fail to process a message | As a user, I'd like to be notified when a exception is thrown in a module so that I can tap into an error channel to receive the failures for each stream/module. | 8 |
| 833 | XD-924 | Split integration.x in dedicated XD projects where appropriate | Similar to what is done for e.g. hadoop, reactor, and http, some of the classes in the .x package (namely gemfire, splunk, twitter) should go in dedicated (albeit small) projects. This would enable further modularization (see XD-915) | 3 |
| 834 | XD-925 | Investigate Swagger to generate REST API Documents | NULL | 3 |
| 835 | XD-926 | Update Core Spring Dependency to 4.0.0.M3 | NULL | 8 |
| 836 | XD-927 | ModuleType Refactor | Remove ModuleType.getModuleTypeByTypeName. All code should use the enum. | 3 |
| 837 | XD-928 | Refactor src/test/resources in Dirt | * In the testmodules.source ** Rename source-config to packaged-source ** Rename source-config to packaged-source-no-lib * All xml files should be prefixed with test. i.e. testsource, testsink * Make sure all tests pass with new configuration | 1 |
| 838 | XD-929 | Remove the Tabulation characters should not be used from Sonar | NULL | 1 |
| 839 | XD-930 | Return rounded interval values from aggregate counter queries | The aggregate counter query result currently returns the interval that is passed in, whether it is aligned with the bucket resolution requested or not. It would be more intuitive if the time values returned are rounded (down) to the resolution of the query (i.e. whole minutes, hours, days or whatever). | 2 |
| 840 | XD-931 | Format option to display runtime module properties in shell | The runtime module properties requires a format option when displayed in the Shell Based on the PR (https://github.com/spring-projects/spring-xd/pull/340), the module properties are stored as String and displayed as is. | 2 |
| 841 | XD-932 | Tests Fail because HSQL not started | This is cause generally by someone having port 9100(hsqldb port) in use. It is recommended that setup checks to see if port is in use. If it is throw an exception stating that hsqldb port (9100) is in use and suggest options like. Free up the port or change the hsqldb port. | 2 |
| 842 | XD-933 | Remove work around Spring HATEOAS#89 | See https://github.com/spring-projects/spring-hateoas/issues/89 Updating HATEOAS version and removing in a lot of controllers should be possible now. See eg https://github.com/spring-projects/spring-xd/blob/4919ea2498a13ef47aaa9437937308fb26a7a24f/spring-xd-dirt/src/main/java/org/springframework/xd/dirt/rest/XDController.java#L219 | 2 |
| 843 | XD-934 | Support JSON to Object | Register a JSON to Object converter in the DefaultContentTypeAwareConverterRegistry. Currently we only support Object to JSON but not the other way. This may or may not work for an arbitrary object but is useful in many cases. | 2 |
| 844 | XD-935 | Implement Kryo serialization for Tuple | Currently TupleCodec uses JSON for serialization/deserialization. It should use Kryo. This will require some customization and potentially changes to Tuple to address the Tuple's conversionService field. | 3 |
| 845 | XD-936 | Find runtime modules by type and/or name | We need a way to find the runtime module info by module type("source", "sink", "processor", "job"). | 3 |
| 846 | XD-937 | Inject ModuleDefinitionRepository into ModulesController | Currently, ModulesController creates the ModuleDefinitionRepository instance with ModuleRegistry. Instead, we should inject the moduleDefinitionRepository into ModulesController directly. | 1 |
| 847 | XD-938 | List runtime modules by wrong containerId should throw exception | With PR#340, listing of runtime modules with a non-existent containerId will display empty table. Instead, we can throw exception saying Container doesn't exist. | 1 |
| 848 | XD-939 | Make Runtime modules listing by ContainerId pageable | The RuntimeContainersController (from PR#340) returns the list of runtime modules. Instead we need make it pageable. | 2 |
| 849 | XD-940 | Splunk Pulls in an Old SI Jar into STS | NULL | 1 |
| 850 | XD-941 | Convert remaining modules to be CP-aware | Now that XD-924 is merged, we can convert splunk, twitter and gemfire modules to rely on the newly created projects. The hadoop module will get its own story, as classpath handling is a bit more tricky for that one. From there on, no new dependency should be added to DIRT for the sole purpose of a module. Rather, it should directly be created as a CP-aware module and project. | 3 |
| 851 | XD-942 | Change repo links in build.gradle repo.spring.io | NULL | 1 |
| 852 | XD-943 | JobPlugin should not add properties that are not needed | If you create a simple "http | log" stream and list the modules, you'll see that JobPlugin adds its numberFormat, makeUnique, etc properties. Even though they do no harm, it's really strange for users. Plus, it could conflict with e.g. activation of profiles given present properties | 3 |
| 853 | XD-944 | Clean up unused JSON mapping classes | There are a few obsolete classes lurking around, TypedJsonMapper comes to mind but there are likely some others | 1 |
| 854 | XD-945 | Spike for writing to HDFS | See Epic https://jira.springsource.org/browse/XD-234 | 10 |
| 855 | XD-946 | Spike for EC2 deployments | NULL | 20 |
| 856 | XD-947 | Spike for Deployment SPI | SPI for deployment on to YARN + Local 'dirt' cluster. | 20 |
| 857 | XD-948 | Spike for job that imports data from CSV file to HDFS | TBD | 10 |
| 858 | XD-949 | Spike for job that exports HDFS CSV data to JDBC | NULL | 10 |
| 859 | XD-950 | Spike for advanced Job Orchestration features | NULL | 20 |
| 860 | XD-952 | Spring-XD shell can't run commands with kerberized CDH 4.3.0 | Basically when I launch the spring-xd shell I can't interact with the namenode I receive security violations despite the fact that I try and set the proper configs. Authentication/Authorization (true/kerberos) details of this issue can be found here: http://stackoverflow.com/questions/19258321/the-spring-xd-xd-shell-cant-run-the-hadoop-fs-ls-command-the-command-returns | 5 |
| 861 | XD-953 | Richer module options metadata | Module options are currently implemented using PropertySourcesPlaceholderConfigurer, doing simple "text" replacements. This story proposes to introduce a richer model for module options, which could take the following xml representation (keeping in mind that there would be an underlying set of classes that could also apply to e.g. @Configuration approach): {code:xml} <xd:params> <xd:param name="port" help="the http port to listen on" /> </xd:params> {code} So, the very first obvious benefit is providing help metadata. The second one is an easy way to list available option names, without resorting to brittle PropertySourcesPlaceholderConfigurer hacks (we wouldn't for example detect xd.stream.name or XD_TRANSPORT as a valid option) One could also specify defaults/computations, this time benefiting from SpEL everywhere: {code:xml} <xd:params> <xd:param name="directory" default="headers.directory" /> </xd:params> {code} Lastly, this opens the door to better type checking / combinations / optionality support: {code:xml} <xd:params valid="fixedDelay || cron"> <xd:param name="fixedDelay" type="int" /> <xd:param name="cron" type="string" /> </xd:params> {code} The 'valid' attribute can e.g. be evaluated by SpEL. Some final remarks: - That construct can be compatible with our current approach by behaving as a PropertySource itself (instead of creating a PropertySource, the StreamPlugin would give this new bean the java.util.Properties) - Plugins could benefit from a hook in that construct and advertise the fact that they expect/provide new options (e.g. --inputType) There is a slight problem though, which is that if this construct lives in the same AppContext as the module definition itself, then the AppContext needs to be refreshed for the logic to kick in. One could circumvent that using profiles, or we could rely on a different filename convention (e.g. <module>-params.xml) | 10 |
| 862 | XD-954 | Stream definition completion REST layer + Shell adapter | Provide the infrastructure for HTTP GET /completions?start='http | file --d" that would return a list of possible completions (in this case returning the file option names that start with "d") This story is about (and only about): - Having that REST controller, delegating to some "CompletionsEngine" - Implementing the Spring Shell Converter that talks to that It's an empty shell, useless (but easy to do) without the actual "CompletionsEngine" | 3 |
| 863 | XD-955 | Update Jobs documentation to include "job launch" command | This is currently missing and probably supersedes some of the stuff that's in there now. | 1 |
| 864 | XD-956 | Refactor DSL parser according to latest syntax proposals | We have recently revised the syntax for stream definitions, this issue covers that refactoring. | 12 |
| 865 | XD-957 | Update SI Dependency to 4.0.0.BUILD-SNAPSHOT | NULL | 3 |
| 866 | XD-958 | Build failure on Ubuntu | Build fails: gradle clean build ... :spring-xd-dirt:test org.springframework.xd.dirt.stream.FileSourceModuleTests > classMethod FAILED java.io.IOException at FileSourceModuleTests.java:53 org.springframework.xd.dirt.stream.FileSourceModuleTests > classMethod FAILED java.lang.IllegalArgumentException at FileSourceModuleTests.java:130 328 tests completed, 2 failed, 11 skipped :spring-xd-dirt:test FAILED FAILURE: Build failed with an exception. Looks like it is trying to create a directory under the local filesystem / java.io.IOException: Unable to create directory /tmpfilesourcetests org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:2024) org.springframework.xd.dirt.stream.FileSourceModuleTests.createTempDir(FileSourceModuleTests.java:53) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.junit.runners.ParentRunner.run(ParentRunner.java:309) org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:80) org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:47) org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:49) sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) com.sun.proxy.$Proxy2.processTestClass(Unknown Source) org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:103) sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355) org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) | 3 |
| 867 | XD-959 | Running gradle idea creates project configured with source 1.6 | The gradle idea task creates a project configured with source 1.6 This results in compile failures on Java 7 specific code | 3 |
| 868 | XD-961 | Accessing xd-admin URLs in the browser return XML and not JSON | Here is an example: the following request for streams: http://ec2-23-20-25-30.compute-1.amazonaws.com:9393/streams Returns: This XML file does not appear to have any style information associated with it. The document tree is shown below. <errors xmlns:atom="http://www.w3.org/2005/Atom"> <error logref="HttpMessageNotWritableException"> <message> Could not marshal [PagedResource { content: [links: [<http://ec2-23-20-25-30.compute-1.amazonaws.com:9393/streams/ticktock>;rel="self"]], metadata: Metadata { number: 0, total pages: 1, total elements: 1, size: 20 }, links: [] }]: null; nested exception is javax.xml.bind.MarshalException - with linked exception: [com.sun.istack.SAXException2: unable to marshal type "org.springframework.xd.rest.client.domain.StreamDefinitionResource" as an element because it is not known to this context.] </message> </error> </errors> | 1 |
| 869 | XD-962 | fix gradle clean test | Currently ./gradlew clean test fails since the module dependencies are not packaged before the test task. | 1 |
| 870 | XD-963 | Features and bug fixes for named channels | NULL | 10 |
| 871 | XD-964 | Support for common use cases using quartz | NULL | 40 |
| 872 | XD-965 | Low story point/time improvements or bug fixes that can easily be picked up in a sprint in an ad-hoc manner | NULL | 20 |
| 873 | XD-966 | User requests that XD team members are issued EC2 accounts | * Setup groups for xd user * Setup privileges so users can only see their instances * Setup user accounts ** Send created access key to users ** Send username and passwords to user | 5 |
| 874 | XD-967 | User requires wiki page on how to use the XD EC2 Installer | Add instructions to github wiki on the usage of the installer | 5 |
| 875 | XD-968 | Create google doc with instructions on managing EC2 Instances | * Logging into your XD Account For Example: https://946513944028.signin.aws.amazon.com/console * Discuss how to terminate running instances. ** Users can terminate all instances using the UI on the EC2 admin page * Usage monitoring via CloudWatch * Investigate what metadata in each instance is required so that cloudwatch can track | 3 |
| 876 | XD-969 | Add functionality to provision EC2 instance and mount EBS | * The Application should ** Create a Spring Application Context. ** Should take a command line parameter --config that will point to a property file with key/value pairs specified below. By default the file xd-ec2.properties will be looked for on the classpath ** Associate EBS shared volume for each machine instance * Packaging ** Use gradle application plugin to generate a ‘bin’ directory with a script to start the application. Seehttp://www.gradle.org/docs/current/userguide/application_plugin.html ** The config file xd-ec2.properties should be in a directory (ideally a 'config' directory parallel to 'bin' and will be loaded by the application by default - loading via the CP is probably the easiest way. ** Create a POJO to easily reference these properties, vs. using a raw java Properties object. How to verify it works * Integration testing ** Create JUnit based tests. JClouds itself has extensive testing, can look at those for structure. ** Verify what you created has been installed. ** Verify ports are open Instance Information ** EBS of 50GB Base for each instance. * Report successful and failed Instances. Key-Value pairs in configuration file properties may include: * cluster-name= a name describing the cluster you are creating * aws-access-keys= the access key assigned to you by admin * aws-secret-key= the secret key assigned to you by admin * private-key-file= the private key file assigned to you by admin. Used for ssh-ing into * multi-node=true/false if true then installer will run the number of nodes as enumerated in the number-nodes property value. if false the installer will start a single node server * number-nodes=The number of nodes(containers) to deploy for this cluster. Value is an integer > 0 * machine-size=The size of machines to be assigned for admin and nodes. small, medium, large * redis-port=6379 * rabbit-port=5279 * xd-dist-url=The url to download the XD to install. for example: http://blahblahblah.zip * ami = The ami image to use for your cluster. for example: ami-dfadsfdadf | 10 |
| 877 | XD-970 | Install XD admin instance on EC2 | *Update the Deployer class to add the following methods * ** RunningInstance deployAdminServer * The install script steps: ** Using XD-977 install distribution ** Setup XD_HOME variable ** start up redis and rabbit using ports as specified in xd-ec2.properties on admin server ** Create configurator directory ** Copy the configurator to containers ** Use port watch to make sure they started ** start admin server ** use port watch to make sure the admin started on 9393 ** Report if admin server started. If it didn't start abort install. ** Report public DNS name of admin-server * Integration Testing ** Verify XD admin has been started *** Create a basic stream (trigger>log)and make sure we get a success code from xd admin was received. *** Query the redis to see if the stream was created. | 10 |
| 878 | XD-972 | Create CI for XD-EC2 project | Automated test will use directly use the Deployer class * asserts on basic info of RunningInstance ** check that EBS was mounted ** that application was unzipped ** redis and rabbit are running via port checks * http requests on admin port for ** root path ** list of modules * @AfterClass that will look for the cluster name and terminate all instances Look at ‘live’ tag in JClouds tests for some additional tactics | 5 |
| 879 | XD-974 | The HDFS Sink should support compressing files as they are copied | Get a java.io.File and copy it into HDFS. Could be text or binary. Write compressed with Hadoop and third party codecs see: (XD-277, XD-279) should initially support: - bzip2 - LZO | 8 |
| 880 | XD-975 | Users need the ability to provision an XD cluster on EC2 via command line tool. | we need the ability to run an XD cluster to get a handle on general issues and missing features based on running the system in a 'true' clustered environment. We don’t need to make this an end-user facing feature in the short term, e.g. set a few keys in the shell and then install via a shell command. | 40 |
| 881 | XD-976 | Bootstrap Project for XD AWS Installer | * Create a Spring application context. (XML or Java, dealers choice…) * Use XD eclipse code format policy * Create Source Package structure * Create Test Package Structure * Gradle build * JClouds and Spring deps | 5 |
| 882 | XD-977 | Add functionality to download XD distribution zip to each EC2 instance as specified by user | * User specifies download distribution zip file from properties file. See XD-969 for key-value pairs * Copy the distribution from the shared EBS/S3 the ebs volume assigned to the each node/admin instance . ** If not on shared ebs/s3 pull from http site specified by user and put on the shared EBS volume. ** Unzip the distribution from on the ebs volume for the instance to the /home/ubuntu directory ** make sure privileges are set to ubuntu not root. How to verify it works * Create a JUNit style integration test that ** Deletes a known .zip distribution from EBS/S3. ** Invoke the application functionality (should be a 1 liner) that will start up the instance and download the .zip distribution from the URI provided in xd-ec2.properties. Verify the file is now in EBS/S3 and also on the instance ** Tear down created instance ** Create new instance passing in the same URI of the .zip distribution. Verify as before | 5 |
| 883 | XD-978 | Installer needs to launch a single node XD instance | * Create a Deployer class has methods ** RunningInstance deploySingleNode *** takes into account machine size as specified in properties file ** void destroyAllInstances() *** or whatever JClouds returns from the destroy call ** ctor gets passed in the root boostrapping credentials. * Install Script Steps ** Setup XD_HOME variable ** Make sure privileges are set to ubuntu not root. ** start up redis and rabbit using ports as specified in xd-ec2.properties ** Use port watch to make sure they started ** Start singlenode after configuration. ** Display hostname of singlenode server ** Report successful and failed startup ** Hit root of xd-admin to see if there is a response on 9393 * Integration Testing ** Verify that config files have been setup ** Verify XD has been started ** Verify XD can process a basic http post | 10 |
| 884 | XD-979 | Batch Wordcount Sample to use File Source | NULL | 3 |
| 885 | XD-980 | Install XD Container on EC2 | *Update the Deployer class to add the following methods* * RunningInstance deployContainer * For each of the containers: ** Using XD-977 install distribution ** Setup XD_HOME variable ** Create configurator directory ** Copy the configurator to containers ** Run Configurators ** Verify that configuration files are setup correctly ** start container server ** Report if container started. (using jmx/Jolokia, if available) If it didn't start report failure but continue. ** Report public DNS name of container * Integration Testing ** Verify that config files have been setup properly ** For each container: *** Verify container has been started *** Verify that container is working by creating a stream in admin (trigger|log). Verify that the log on the container is being updated. | 10 |
| 886 | XD-981 | Missing guava-11.0.2.jar dependency for hadoop distros | We used to have a shared guava-11.0.2.jar dependency in the lib dir. That's no longer there so hadoop distros that require this now fail (at least any hadoop 2.0.x based ones) We should also upgrade to current Hadoop versions (Hadoop 2.2 stable) | 3 |
| 887 | XD-982 | OOTB batch jobs for common cases | NULL | 20 |
| 888 | XD-983 | Writing to HDFS - 1.x | NULL | 1 |
| 889 | XD-984 | Create OOTB batch job for import and processing multiple files to JDBC | Create a sample batch job for inclusion in the distribution that will perform the following tasks. ItemReader * Read a from a directory with multiple files (configurable) * Support for CSV (assume first line has header values) * Convert to tuple data structure ItemProcessor * Provide groovy based no-op ItemProcessor. (configurable) ItemWriter * Write to JDBC. ** Provide ** Assume there is an DB instance running somewhere, specify connection info (configurable) **The sample job should be documented** | 10 |
| 890 | XD-985 | Create OOTB 'file to HDFS' batch import job that is launched by a stream. | Same processing as XD-984, but the job instacne is launched via an event from the input file source. Supporting a single file per job launch is OK. > job create blah --definition "filehdfs" > stream create csvStream --definition "file --ref=true --dir=/Users/luke --pattern=*.csv > job:blah" *the job should be documented* | 5 |
| 891 | XD-986 | Create OOTB batch job that uses the Hadoop Shell to copy multiple files from a local directory into HDFS | Create a new tasklet implementation that will use the ANT globbing like syntax found in MultiItemResourceReader to specify the input files, and use FsShell to invoke a 'copyFromLocal' command with a specified HDFS output directory. The batch job would call the tasklet. *the job should be documented* | 10 |
| 892 | XD-987 | Create OOTB batch job for export and processing multiple files from HDFS to MongoDB | The ItemReader will read multiple files from HDFS and the data will be converted to a tuple data structure The ItemProcessor will be a no-op groovy script. The ItemWriter will write the data to a MongoDB collection ** A TupleToDBObject converter will need to be developed. *the sample job should be documented* | 8 |
| 893 | XD-988 | Create OOTB batch job for export and processing multiple files from HDFS to JDBC | Same setup as XD-987 for ItemReader and ItemProcessor, but should write to HDFS. One can assume that the table structure has been created already external to the batch job execution. | 5 |
| 894 | XD-989 | Create OOTB batch job that uses the Hadoop Shell to copy multiple files from HDFS to a local directory | This is the inverse of XD-986 and will require creating a custom tasklet but with the input/output reversd. | 5 |
| 895 | XD-990 | The HDFS Store Library should support writing text with delimiter | Support writing lines of text separated by a delimiter Support writing a CSV (comma-separated variables), TSV (tab-separated variables), No compression | 8 |
| 896 | XD-991 | The HDFS Store Library should support compression when writing text | Need to support writing text in compressed format should initially support: - bzip2 - LZO | 8 |
| 897 | XD-992 | The HDFS Store Library should support writing to Sequence Files | Support for writing Sequence Files Without Compression Need a means to specify the key/value to be used | 8 |
| 898 | XD-993 | The HDFS Store Library should support compression when writing to Sequence Files | Support for using compression when writing Sequence Files Either block or record-based compression. | 8 |
| 899 | XD-994 | The HDFS Sink should support writing POJOs to HDFS using Parquet | Writing POJOs using Kite SDK | 8 |
| 900 | XD-995 | Refactor tests with FileSink|FileSource to use eventually() matcher | Some tests (esp. ModuleClasspathTests.testModuleWithClasspathAfterServerStarted) seem to fail because of a race condition. Add a Hamcrest matcher that knows how to read the content of a FileSink|Source and refactor those to read like e.g. assertThat(fileSink, eventually(hasContent("foo))) | 5 |
| 901 | XD-997 | The HDFS Sink should support writing POJOs to HDFS using Avro/Kite SDK with support for partitioning | Support for partitioning on a field, e.g. date. | 4 |
| 902 | XD-998 | Add documentation for gemfire cache-listener source | Need some sample usage, docs for https://github.com/spring-projects/spring-xd/tree/master/modules/source/gemfire | 1 |
| 903 | XD-999 | Return the step execution information in the current job execution controller | Related to https://jira.springsource.org/browse/BATCH-2109 DistributedJobService#listJobExecutionsForJob overrides SimpleJobService#listJobExecutionsForJob and does not include the *StepExecution*s. This is due to serializion issues with Jackson. In order to fix this, we need to add a Jackson MixIn. | 4 |
| 904 | XD-1000 | User should be able to view the list of all the job definitions | Create a tab view with tabs “Job Definitions�?, “Runtime Jobs�? (Deployed Jobs?, “Job Instances�? (Is Runtime Jobs a better name here?) and “Job Executions�?. On clicking “Job Definitions�? tab, we can have a table view of job definitions. Since we bootstrap.js, we can have a responsive table layout to list all the available job definitions. At the REST layer, “/jobs�? provides the list of job definitions. We can expand the JobsController list()’s QueryOptions to add more criteria (especially to list JobDefinition’s status (Deployed/Undeployed). Also, this is the UI implementation for the shell command “job list�? | 4 |
| 905 | XD-1001 | User should be able to view the list of all Deployed Jobs | On clicking the “Deployed Jobs�?, we can have a table view of all the deployed jobs. This is again a responsive table layout with all the job definitions with status “deployed�?. The deployed XD job corresponds to a single batch Job Instance. This story addresses the UI layout changes to display existing JobInstance information. | 2 |
| 906 | XD-1002 | UI: User should be able to get all the job executions on a given job at deployed jobs page | On clicking a specific job name in the deployed jobs list, we need to redirect the user to show the list of all the job executions on that job. User should be able to navigate back to the deployed jobs list. | 3 |
| 907 | XD-1003 | UI: User should be able to launch a job from Deployed jobs page | From the Deployed jobs page, user should be able to click on the "Launch" button on a specific job and specify the Job parameters as key value pairs in the text box and we will convert that into JSON string as JobParameters into JobLaunch request. | 2 |
| 908 | XD-1004 | UI: User should be able to schedule a job from Deployed jobs page | From the Deployed jobs page, by clicking the "Schedule" button on a specific deployed job row, user should be able to schedule this job with: 1) Cron trigger (with cron expression) as a source to job launching named channel 2) Fixed rate/delay trigger as a source to job launching named channel | 4 |
| 909 | XD-1005 | UI: User should be able to filter the list of executions on the execution tab | On clicking the “Executions” tab, user should see the list of all batch job executions. There should be options to filter job executions by few criteria such as by “Job name”, “execution time” etc., | 3 |
| 910 | XD-1006 | UI: User should be able to view job detail from a specific job execution at Job Executions page | On clicking "details" link on a job execution row, user should see the job details. Job detail page will show all the information about the job, where as the table listing of jobs on the Execution tab may have omitted some columns or aggregated values to convey information more easily. | 3 |
| 911 | XD-1007 | UI: User should be able to see step execution info in a table below job detail | On clicking the job detail page, we should display all the step executions associated with the specific job execution in a table view. | 3 |
| 912 | XD-1008 | Jobs list REST endpoint should include deployed/undeployed status | Currently, the jobs definition list REST endpoint doesn't include deployed/undeployed status on a given job. | 4 |
| 913 | XD-1009 | Don't perform error-level logging for normal application behavior in batch admin functionality | Assumption: The batch job *wordCountJob* does not exist. Executing the REST endpoint: http://localhost:9393/batch/jobs/wordCountJob/executions yields the correct response: {code} <errors xmlns:atom="http://www.w3.org/2005/Atom"> <error logref="NoSuchBatchJobException"> <message> Batch Job with the name wordCountJob.job doesn't exist </message> </error> </errors> {code} One could argue, though, that this is technically not an error but a status/application message, no?? However, the console/log logs at error level with a full stacktrace. I think the full stacktrace should only be available when debugging is allowed. In normal mode, I think we should log only at info or warn level without the full stacktrace. | 2 |
| 914 | XD-1010 | Convert remaing FileSink assertions to eventually() constructs | See discussion at https://github.com/spring-projects/spring-xd/pull/362 | 5 |
| 915 | XD-1011 | Spring Shell --host option does not work | When trying to use the --host command line parameter, spring shell reports the following error for this command, "xd-shell --host localhost" : Oct 25, 2013 10:35:25 AM org.springframework.shell.core.SimpleParser commandNotFound WARNING: Command '--host localhost' not found (for assistance press TAB) | 2 |
| 916 | XD-1012 | DataSource for batch infrastructure should be configurable, not hardcoded to hsqldb | Currently HSQLDB is the only option for batch jobs. This should be configurable so that a user may select another JDBC data store option. Steps: * hsqldb.properties needs to be changed to batch-jdbc.properties * hsql prefixes should be changed to batch-jdbc * JDBC Connection String needs to be configurable * JDBC Driver needs to be configurable * The Setup Scripts to be used for spring batch need to be configurable. * HSQLServerBean should be renamed to something batch-jdbc.properties Tests: Should be able write tests that support HSQLDB | 8 |
| 917 | XD-1013 | Add additional REST endpoint that return the XML definition of a job | NULL | 8 |
| 918 | XD-1014 | Command to show the XML of the job definition | NULL | 8 |
| 919 | XD-1015 | Create a reusable responsive UI layout to render the XD PagedResoures in tabular view | Create a reusable responsive UI layout to render the PagedResources returned from REST endpoint. As part of this, try upgrading the bootstrap to 3.0.0 and use the responsive styles offered in it. | 5 |
| 920 | XD-1016 | Provide an option to pretty print JSON output | Probably the cleanest approach is to provide a properties file in the xd config directory that enables this globally, e.g., json.pretty.print=true. This will require some refactoring of the ModuleTypeConversion plugin, i.e., use DI in streams.xml | 3 |
| 921 | XD-1017 | Spring Batch Admin UI looks to localhost when getting status updates | Currently when you bring up a admin-ui on a browser that is not on the XD admin server, the page reports an error. This is because the javascript is looking to localhost to get its updates. | 5 |
| 922 | XD-1018 | JobDeployer hides root exceptions on failure | When deploying jobs, the following code (line 103), hides the root cause of deployment failure: if (exceptionClassName.equals(BEAN_CREATION_EXCEPTION) || exceptionClassName.equals(BEAN_DEFINITION_EXEPTION)) { throw new MissingRequiredDefinitionException(definition.getName(), cause.getMessage()); } For example: org.springframework.xd.dirt.stream.MissingRequiredDefinitionException: Error creating bean with name 'dataSourceInitializer' defined in file [/Users/luke/Work/spring-xd/build/dist/spring-xd/xd/modules/job/hdfsjdbc/config/hdfsjdbc.xml]: Cannot resolve reference to bean 'databasePopulator' while setting bean property 'databasePopulator'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'databasePopulator' defined in file [/Users/luke/Work/spring-xd/build/dist/spring-xd/xd/modules/job/hdfsjdbc/config/hdfsjdbc.xml]: Initialization of bean failed; nested exception is java.lang.NullPointerException org.springframework.xd.dirt.stream.JobDeployer.deploy(JobDeployer.java:103) org.springframework.xd.dirt.stream.JobDeployer.deploy(JobDeployer.java:67) org.springframework.xd.dirt.rest.XDController.save(XDController.java:242) | 1 |
| 923 | XD-1019 | Add Spring Retry to Rabbit Message Bus | Log and purge bad messages | 2 |
| 924 | XD-1020 | Cleanup ModuleDeployer | ModuleDeployer has many methods with very similar names that are hard to understand. Moreover, there is a substantial amount of duplicated code that should be extracted in sub methods with descriptive names. One should even consider splitting the class | 5 |
| 925 | XD-1021 | Fix undeploy of stream with a composed module | Create a composed module Create a stream that uses that module Try to undeploy the stream. Kaboom The dispatch is not correctly implemented in ModuleDeployer | 5 |
| 926 | XD-1022 | Switch "module list" to horizontal display | The module list command currently has a very simplistic two column display of (module name, module type). The is not very readable. Switch to a 4 column display: (Sources, Processors, Sinks, Jobs) Additionally, mark composed module [e.g. "myhttp (c)"] | 5 |
| 927 | XD-1023 | Refactor job deploy to go through XDController.deploy(name) | Job deployment currently goes through an overloaded version of deploy() that takes 4 parameters. This prohibits job handling code from benefiting from common behavior (and eg currently breaks deployAll) Given that the 3 additional parameters are in fine handled as module parameters, let's push them to the job definition, known at creation time, rather than at deployment time (as it does not really make sense to change those between deploys) | 4 |
| 928 | XD-1024 | Kryo Redis Serializer | The current (XD) uses an inadequate {{MessageRedisSerializer}} when {{extractPayload}} is false (not currently being used). When porting this to SI, the serializer will be dropped. Suggest creation of Kryo serializer for Messages for when Redis source/sinks are created. | 5 |
| 929 | XD-1025 | UI: Implement Job Deploy/Undeploy from the Job Definitions page | From XD-1023, the job status(deployed/undeployed) is available from JobInstance Repository and a job can be deployed/undeployed correctly. Implement Job deploy/undeploy for a given job from JobDefinitions page and indicate status of the job definition (deployed/undeployed). | 2 |
| 930 | XD-1026 | UI: Responsive layout for supported user agents(Mobile, Tablet and Desktop) | The current admin UI uses bootstrap 3.0.0 which provides responsive design. We need to expand our scope to support all supported user agents. This requires changes to use user agent specific layout for the UI views. | 10 |
| 931 | XD-1027 | Create script-based batch ItemProcessor | This would be included in the OOTB batch jobs to optionally process the loaded Tuple with a configured script. | 10 |
| 932 | XD-1028 | UI - Do not hard-code server url | Specify the default URL as: {code} urlRoot: location.protocol+'//'+location.hostname+(location.port ? ':'+location.port: '') {code} | 1 |
| 933 | XD-1029 | Migrate to SI Redis Queue and Topic Adapters | NULL | 3 |
| 934 | XD-1030 | Refactoring to JdbcMessagePayloadTransformer to extends from AbstractPayloadTransformer not JsonToObjectTransformer | NULL | 1 |
| 935 | XD-1031 | UI - Launch a job with parameters | Use a modal dialog to specify runtime parameters. There should be a little text are that gives hints as to the spring batch parameter key/value conventions, e.g. for type. Might be a good idea to have a checkbox that lets you select to 'auto increment' job instance number. 4 columns key, value, type, identifying and an 'add parameter' button that adds a new row. This would appear as a modal dialog box, polling of the state of the deployments would be suspended while the job parameter modal dialog box is shown. | 8 |
| 936 | XD-1032 | Convert hadoop module to isolated classloader scheme | * rename spring-xd-extension-hdfs to something else, as it seems it is all spring "data" stuff and is not coupled to xd. But leave it in extensions/ for now * rename and move spring-xd-hadoop inside extensions (maybe to spring-xd-extension-hadoop (or hdfs)) * make hadoop related modules depend on the latter (which itself will depend on the former) | 8 |
| 937 | XD-1033 | Add a tcp-client source module | Add a module that can act as a tcp *client* (as opposed to our current tcp module, which acts as a server, waiting for an incoming connection) Also, the module should allow to send "commands" to the remote server. The typical minimal case for such a protocol is to send "PING" messages, but a stateful mechanism should be put in place for more complex cases. | 5 |
| 938 | XD-1034 | Stream definitions should not be saved when auto-deploy results in an error | This relates to XD-871 which provides a good scenario. >stream create s1 --definition "http | log --deploy" >stream create s2 --definition "http | log --deploy" The second command results in an error message that the port is in use but the stream definition is still saved. Since create + deploy is a logical unit of work, it should follow transactional semantics. In other words if the deploy fails, the repository should be rolled back (or a compensating destroy should be performed). Note this should not be handled the same way if create and deploy happen separately. In that case, the stream definition should remain. | 4 |
| 939 | XD-1035 | Attempt to compose a module with same name+type as existing should fail | As per discussion, any attempt to create something that would collide with existing should fail: II) Attempt to create a composed module with same name as an already existing, not composed module, with same type should fail (EB, MF) should work and shadow previous module from now on III) Attempt to create a composed module with same name as an already existing composed module, with same type should fail (EB, MF) should work and shadow previous module from now on should work and retroactively change definitions of streams with that module | 5 |
| 940 | XD-1036 | Support composed module deletion | Provided it is not currently used in any stream: V) Attempt to destroy a composed module Should not be supported at all Should not be supported if involved in at least one stream (EB?, MF!) Should be supported and have no other consequences whatsoever (see IV) (EB?) Should be supported and invalidate/destroy streams involving it | 10 |
| 941 | XD-1037 | Support ModuleOptions for composed modules | With the following semantics (assuming http | filter | transform composition): * Fully qualified names always available (eg filter.expression) * using module type as key * or label if ambiguity * Simple names available if no ambiguity (eg here "port" refers to http.port) As per discussion: VIII) Use of “--expression�? option in composed module “filter | filter�? Should not be supported (require FQN always) Should pertain to first filter module (always) Should pertain to all modules with an “expression�? option (here: both filters) Should fail if not qualified and ambiguity. Use label.expression (or type.expression, eg if we had “filter | transform) (see VII ⇒ nesting1.nesting2.nesting3.expression) (MF) VIII) Use of “--expression�? option in composed module “http | filter --expression=something | transform�? Should not be supported (already valued) Should pertain to first filter module (always) Should pertain to all modules with an “expression�? option (here: both filters) Should fail if not qualified and ambiguity. Use label.expression (or type.expression, eg if we had “filter | transform) (see VII ⇒ nesting1.nesting2.nesting3.expression) (MF) | 15 |
| 942 | XD-1038 | Source that produces JSON String as byte[] with --outputType application/json should produce JSON string | NULL | 2 |
| 943 | XD-1039 | Composed of Composed fails at stream deployment time | Although composition of a module out of an already composed module seems to work at the 'module compose' level, trying to deploy a stream with that more complex module fails with org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:724) Caused by: java.lang.IllegalArgumentException: each module before the last must provide 'output' org.springframework.util.Assert.notNull(Assert.java:112) org.springframework.xd.module.CompositeModule.initialize(CompositeModule.java:132) org.springframework.xd.dirt.module.ModuleDeployer.deploy(ModuleDeployer.java:234) org.springframework.xd.dirt.module.ModuleDeployer.deployModule(ModuleDeployer.java:224) org.springframework.xd.dirt.module.ModuleDeployer.handleCompositeModuleDeployment(ModuleDeployer.java:180) org.springframework.xd.dirt.module.ModuleDeployer.handleMessageInternal(ModuleDeployer.java:129) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) ... 63 more | 5 |
| 944 | XD-1040 | Remove "legacy" application code following Spring bootification | See the discussion in https://github.com/spring-projects/spring-xd/pull/370 There are now various superseded classes and tests which we no longer need. | 2 |
| 945 | XD-1041 | Upgrade to Spring for Apache Hadoop 1.0.2.RELEASE and Pivotal HD 1.1 | Make sure the sinks and jobs work against Pivotal HD 1.1 | 3 |
| 946 | XD-1042 | Support short names for types in ModuleOptions | The ModuleOptions PR currently uses FQN for types (eg java.lang.String) Would be nice to have support for short names for common types, both in the properties files and the annotation | 3 |
| 947 | XD-1043 | Encapsulate list of ModuleDeploymentRequests within a parsed Stream result object | Currently the parser returns a List<ModuleDeploymentRequest>, and the deployer works with that list directly. We need a higher level parser result (e.g. DeployableStream - or probably a better name after some thought) that can encapsulate that list while also enabling metadata to be added. That metadata may be helpful for composite module information as well as the module dependencies of a given stream (including any composed modules within that stream). | 8 |
| 948 | XD-1044 | refactor module dependency tracking to be closer to stream deployment | see comments on this PR (which is part of the code that needs to be refactored): https://github.com/spring-projects/spring-xd/pull/390 | 8 |
| 949 | XD-1045 | Create project for model that is common between client and server | this would elminate dependencies that are currently in the codebase, such as: * RESTModuleType and ModuleType enums * ModuleOption and DetailedModuleDefinitionResource.Option | 5 |
| 950 | XD-1046 | Consider a shared project for POJOs that are shareable betweed model and REST layer | NULL | 4 |
| 951 | XD-1047 | Allow Aggregate Counter to use timestamp field in data. | Currently the aggregate counter aggerates by the current time. However the data may already have a timestamp in it (eg streams from activity events on a website). It would be useful as an alternative approach to be able to specify this field to aggregate on. This would have the following benefits: 1) The aggregate counts would be more accurate as they would reflect the acutal event times and not have any lag from an intermediate messgaging system they might have passed through. 2) If for whatever reason XD is down, comes back up and starts pulling queued messages from the messaging system the aggregate counter will reflect the correct event time. Currently you would get a gap and then a spike as a backlog of messages would get allocated to the current aggregate count. 3) Old data could be rerun through XD still creating the correct aggregate counts. Configuration would be something like stream create --name mytap --definition "tap:mystream > aggregatecounter --name=mycount --timestampField=eventtime" without the timestampfield it would behave as currently. | 5 |
| 952 | XD-1048 | Extend aggregate counter to dynamically aggregate by field values in addition to time. | This would be a combination of the existing aggregate counter and field value counter functionality. For example if the stream data was for car purchases some fields might be colour, make and model. When analysing the aggregate data I dont just want to know how many were sold on Monday, but how many of each make or how many of each colour, or how many of a particular colour, make AND model. This would allow a dashboard type client to 'drill down' into each dimension or combination of dimensions (in real time without executing batch queries against the raw data) Ideally the aggregate counter would be specified as stream create --name mytap --definition "tap:mystream > aggregatecounter --name=mycount --fieldNames=colour,make,model" The keys would be dynamically created according to the field values in each record (ie in a similar way to the field value counter you would not need to predefine field values) and keys would be created for all combinations of the fields specified eg the record { "colour":"silver" , "make":"VW" , "model" : "Golf" } would increment the following key counters (in addtion to the existing time buckets) <existing base counter - ie all fields for this time bucket> colour:black make:VW model:Golf colour:black.make:VW colour:black.model:Golf make:VW.model:Golf colour:black.make:VW.model:Golf ie the actual keys would look something like aggregatecounters.mycount.make:VW.model:Golf.201307 etc This may seem like it would generate a lot of key combinations but in practice the data generated will still be massively less than the raw data, and keys will only be created if that combination occurs in a time period. Also some fields may be dependent on each other (such as make and model in the above example) so the amount of possibilites for those composite keys would be a lot less that the number of one times the number of the other. | 5 |
| 953 | XD-1049 | Create documentation for composed modules | NULL | 3 |
| 954 | XD-1050 | Improve Module Options support | note from PR #365 - which has been merged - providing the initial level of support... Pending issues (to be addressed in another PR?): - [x] complex case - [x] default values for complex case, when option is not surfaced back to the module (eg "suffix" in our canonical example) - [ ] plugin provided options and values - [ ] descriptive defaults instead of actual defaults (e.g. \<use stream name\>) - [ ] JSR303 Validation | 12 |
| 955 | XD-1051 | Rename or reconsider the "module display" command | note from PR #365: {quote} We should probably create a new story to (at least) rename "module display" to "module showconfig" or something, but possibly even reconsider it altogether. In some sense, it's even violating the encapsulation of ModuleRegistry. It wont work for java-config style modules or spring-integration Groovy DSL modules. Personally, if anything, I'd rather see those config files themselves exposed as part of an admin UI. What you are doing with the options here fits better with the encapsulation principle and the fact that typical usage should not require detailed knowledge of the actual underlying configuration of a Module. {quote} | 1 |
| 956 | XD-1052 | Enforce consistent naming across CLI options, and command/template/operations method names | e.g. see comment on PR #390: https://github.com/spring-projects/spring-xd/pull/390/files#r7563787 In that case, it's "delete" in one place and "destroy" in another. There are other cases as well. | 4 |
| 957 | XD-1053 | Add "http get" command to shell | This will allow for some easy demonstration of how 'HATEOS' works via links in our REST API. There are probably some quite useful commands here that could be used from the Spring Data REST shell longer term, but a good place to take a look at now. Goal is to show how metrics such as counters are accessible w/o having to switch tabs to use wget. The pretty printing of the returned JSON is an important feature to help understand the response, this functionality can be taken from/reused from the Spring Data REST shell | 3 |
| 958 | XD-1054 | Deployment tab on admin-ui lags behind with changes on other pages | The deployment tab does not reflect the current state of the jobs. User must hit browser refresh button to make it work. | 4 |
| 959 | XD-1055 | Schedule button shows up but does not perform any task | I know that the feature is not ready, but we either should post a message to the user that the feature is not available or hide the button | 5 |
| 960 | XD-1056 | Empty parameter sent to job when launched from UI | Job gets a empty key:value pair when launching the job from the admin-ui. | 5 |
| 961 | XD-1057 | Colors on Job Definition tab are different than other tabs | The table background color on the Job Definition Tab is green while the others tabs have a white background. They should be consistent. | 1 |
| 962 | XD-1058 | Remove unnecessary LESS files from XD UI styles | Currently, the bootstrap.less file has all the styles that the bootstrap supports. But we should only add/compile the LESS that are needed by XD UI. | 2 |
| 963 | XD-1059 | Batch Job's step execution count is always '0' | The batch job's step execution count is retrieved from org.springframework.batch.admin.web.JobExecutionInfo in batch job repository. But, the JobExecutionInfo always have the stepExecutionCount set to '0'. | 2 |
| 964 | XD-1060 | Add support for Hortonworks Data Platform 2.0 | (apologies if a ticket already exists for this, but I didn't see one) I spun up the Hortonworks Data Platform 2.0 sandbox, but see it isn't supported by Spring XD yet. How hard would it be to add these Distro's in? Is it just a matter of dropping in a lib folder for hadoop22 and/or hdp20, and allowing those and options to be passed in via the --hadoopDistro option? I'm currently trying to work through the following tutorial, but using the HDP 2.0 sandbox instead of the 1.3 sandbox http://hortonworks.com/hadoop-tutorial/using-spring-xd-to-stream-tweets-to-hadoop-for-sentiment-analysis/ Thanks! | 1 |
| 965 | XD-1061 | Upgrade asciidoctor-gradle-plugin to 0.7.0 | Looks like we need to spend a cycle on Asciidoc - as we still have the author-tag-issue - I thought we can simply upgrade the asciidoctor-gradle-plugin to 0.7.0 (currently 0.4.1) but that breaks the docs being generated. | 2 |
| 966 | XD-1062 | User would like to see request/response details (headers URLs etc) in shell | Add debug flag for logging globally maybe or else --trace per command? | 1 |
| 967 | XD-1063 | Fix mqtt module properties | Use of dot in property name prevents the user from specifying a value in stream definition Also, defaults are repeated at .xml and .properties level | 2 |
| 968 | XD-1064 | Update router sink logic to match new channel syntax | for example, the following should work: {code} xd:>stream create a1 --definition "queue:a > transform --expression=payload+'-a' | log" Created new stream 'a1' xd:>stream create b1 --definition "queue:b > transform --expression=payload+'-b' | log" Created new stream 'b1' xd:>stream create s1 --definition "http | router --expression=payload.contains('a')?'queue:a':'queue:b'" Created new stream 's1' xd:>http post --data "ha" > POST (text/plain;Charset=UTF-8) http://localhost:9000 ha > 200 OK // log shows: "ha-a" xd:>http post --data "hi" > POST (text/plain;Charset=UTF-8) http://localhost:9000 hi > 200 OK // log shows: "ha-b" {code} This needs to be tested against all transports (local, redis, and rabbit) | 4 |
| 969 | XD-1065 | Extra "job" queues being created for all streams | This is most likely an issue for any transport, since it's probably happening within the JobPlugin, but I noticed when using Rabbit that every stream I created also triggered creation of "job:[streamname]" Queues. | 2 |
| 970 | XD-1066 | Topic channels are not broadcasting | This needs to work for all transports (local, rabbit, and redis), and we need to ensure that we have test coverage for each of those to avoid any regressions. The incorrect behavior was observed with all three transports: {code} xd:>stream create a --definition "topic:foo > transform --expression=payload+'-a' | log" Created new stream 'a' xd:>stream create b --definition "topic:foo > transform --expression=payload+'-b' | log" Created new stream 'b' xd:>stream create s --definition "http > topic:foo" Created new stream 's' xd:>http post --data hi > POST (text/plain;Charset=UTF-8) http://localhost:9000 hi > 200 OK // only one line in log! {code} | 5 |
| 971 | XD-1067 | Container with redis as transport emits stack trace on shutdown | This is "harmless" (the attempt to brpop continues even after the connection itself has been closed), but ugly: {code} 12:45:15,084 ERROR redisInboundAdapter-redis:queue-inbound-channel-adapter1 inbound.RedisQueueMessageDrivenEndpoint:181 - Failed to execute listening task. Will attempt to resubmit in 5000 milliseconds. org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is com.lambdaworks.redis.RedisException: Connection closed org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:46) org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:36) org.springframework.data.redis.connection.lettuce.LettuceConverters.toDataAccessException(LettuceConverters.java:159) org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:253) org.springframework.data.redis.connection.lettuce.LettuceConnection.bRPop(LettuceConnection.java:1508) org.springframework.data.redis.core.DefaultListOperations$12.inRedis(DefaultListOperations.java:163) org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:51) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:185) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153) org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86) org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:160) org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:105) org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:178) org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:51) org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:286) org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) java.lang.Thread.run(Thread.java:724) Caused by: com.lambdaworks.redis.RedisException: Connection closed com.lambdaworks.redis.RedisAsyncConnection.await(RedisAsyncConnection.java:1079) com.lambdaworks.redis.RedisConnection.await(RedisConnection.java:820) com.lambdaworks.redis.RedisConnection.brpop(RedisConnection.java:101) org.springframework.data.redis.connection.lettuce.LettuceConnection.bRPop(LettuceConnection.java:1506) ... 12 more {code} We should shutdown the consumer gracefully (i.e. before the connection is closed). | 3 |
| 972 | XD-1068 | Remove existing 'purpose built' json processors and ensure all functionality is still available with #jsonPath based SpEL expression based processors | NULL | 2 |
| 973 | XD-1070 | File to HDFS batch job fails due to "/data" directory not available in HDFS | The batch job for File to HDFS will try to check for the default '/data/' directory even if the target directory in HDFS is something else. If the /data directory isn't there, the job will fail. This should be fixed so there isn't a check on the directory that isn't the final HDFS target directory and the target directory should be created if it doesn't exist. | 4 |
| 974 | XD-1071 | Close HDFS file when Batch job ends | The File to HDFS batch job will not close the file being written to in HDFS when the job completes. The ItemWriter for HDFS needs to incorporate functionality that is present in the standard FlatFileWriter, perhaps inheriting from AbstractItemStreamItemWriter | 6 |
| 975 | XD-1072 | Add bridge module | Add a bridge module per XD-956 to support definitions like topic:foo > queue:bar . Convenient for testing for XD-1066 | 1 |
| 976 | XD-1073 | Change default container port from 9000 to something else | Change the default port since it conflicts with the default port for the http source | 1 |
| 977 | XD-1074 | Multiple SLF4J bindings on the classpath | Summary says it all. When starting, we now get {noformat} SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/ebottard/.gradle/caches/artifacts-24/filestore/org.slf4j/slf4j-log4j12/1.7.5/jar/6edffc576ce104ec769d954618764f39f0f0f10d/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/ebottard/.gradle/caches/artifacts-24/filestore/org.slf4j/slf4j-log4j12/1.6.1/jar/bd245d6746cdd4e6203e976e21d597a46f115802/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] {noformat} Most certainly introduced by 1c4817ee60ae9325f6394dcc78aa803c47818546 or 72baec92f2e7bbe860b4a9dc6c994536c1670881 | 2 |
| 978 | XD-1075 | Restore XD Banner | Migrating to boot dropped the XD banner and its info. Can be restored using eg a boot Initializer and removing the default boot banner | 3 |
| 979 | XD-1076 | Create an FTP tasklet to get remote files and put them in the local file system. | This is along the lines of what is in this blog post http://coreyreil.wordpress.com/2012/12/21/spring-batch-creating-an-ftp-tasklet-to-get-remote-files/ Note that there have been some new developments in SI to get at the underlying stream for FTP. The way to test this is to create a new batch job in XD that has this as it's tasklet. Going forward the target file system will also be HDFS. | 16 |
| 980 | XD-1077 | Proof of Concept for module contributions based off Boot | Develop out a Proof of concept for review that allows users to easily create a new module to Spring XD. Key features to be enabled: 1. Allow existing modules to be included as a classpath dependency 2. Consolidate shared libraries in existing modules (e.g. a shared lib for all gemfire modules) 3. User develops a module as a normal Java project 4. User's project can then be added to a runtime container in the same way (as a dependency) Nice to have: 1. at development time container can run continuously and pick up changes from user's module project. 2. Java config for a module. 3. command line functionality to create project scaffolding. | 40 |
| 981 | XD-1078 | Add "spring-xd-exec" directory to Git repo | Without the directory, Spring XD cannot be imported into STS using its own Gradle support. | 1 |
| 982 | XD-1079 | Small method name refactorings and add Javadoc in batch controllers | BatchJobsController.listForJob should be 'executionsForJob' BatchJobsController.jobInstances should be 'instancesForJob' The JavaDoc for the class and each method should be more descriptive about their functionality | 1 |
| 983 | XD-1080 | Make deploy=false as the default when creating a new job | The automatic deployment of the job makes it harder to understand the lifecycle of the job and also does not allow for the opportunity to define any additional deployment metadata for how that job runs, e.g is it partitioned etc. | 1 |
| 984 | XD-1081 | Create documentation on the lifcycle of a job in XD | This should go in the first section of the Batch Job documentation. Here is a rough suggestion The lifecycle of a job in XD. 1. Register a job module with the MoudleRegistry by putting a XML/jar files in the $XD_HOME/modules/jobs directory. 2. Create a job definition from a job module by providing a name and properties that apply to all job instances can be configured at this point. The job is not yet deployed 3. Deploy the job to an xd-container. A job instance can not be created by sending a message to a job queue that contains optional runtime job parameters 4. Launch a job by sending a message to the job queue with job parametres. A Job Instance is created, representing a specific run of the job. A Job Instance is the Job Definition plus the runtime job parameters. You can query for the Job Instances associated with a given job name 5. The job is executed creating a Job Exection object that captures the success or failure of the job. You can query for the Job Executions associated with a given job name. 6. Undeploy a job. This removes the job from xd-container. | 1 |
| 985 | XD-1082 | Create REST API for getting information on a job execution for a given execution id | Adopted functionality from Spring Batch admin Should include springmvc test framework style tests GET /batch/jobs/executions/{executionId} - Get information on all executions of a given job name. | 2 |
| 986 | XD-1083 | Create REST API for getting information on all steps of a given job execution | Adopted functionality from spring batch admin Should include springmvc test framework style tests GET /batch/jobs/executions/{executionId}/steps - Get information on all steps of a given job execution | 4 |
| 987 | XD-1084 | Create REST API for getting information on a given step execution | Functionality adopted from spring batch admin Should include springmvc test framework style tests GET /batch/jobs/executions/{executionId}/steps/{stepId} - Get information on a given step execution | 4 |
| 988 | XD-1085 | Create REST API for stopping a specific job | Functionality adopted from spring batch admin Should include springmvc test framework style tests DELETE /batch/jobs/executions/{executionId} Stop a specific job | 3 |
| 989 | XD-1086 | Create REST API for stopping all job executions | Adopted functionality from spring batch admin Should include SpringMVC test framework style tests. DELETE /batch/jobs/executions/ - stop all job executions | 3 |
| 990 | XD-1087 | Create REST API for restarting a specific job instance | Functionality adopted from spring-batch admin Should include springmvc test framework style tests POST /batch/jobs/{jobName}/{jobInstanceId}/executions - restart a specific job instance | 3 |
| 991 | XD-1088 | Create REST API for getting information on the progress of a given step execution | There is a NPE in the current spring batch admin functionality. Should be springmvc test framework style tests. GET /batch/jobs/executions/{jobExecutionId}/steps/{stepExecutionId}/progress | 4 |
| 992 | XD-1089 | Create shell command for getting information on all job executions for a given name | NULL | 2 |
| 993 | XD-1090 | Create shell command for restarting a specific job instance | NULL | 2 |
| 994 | XD-1091 | Create shell command for getting information on the progress of a given step execution | NULL | 2 |
| 995 | XD-1092 | Create shell command for getting information on all steps of a given job execution | NULL | 2 |
| 996 | XD-1093 | Create shell command for getting information on a given step execution | NULL | 2 |
| 997 | XD-1094 | Create shell command for stopping a specific job | NULL | 2 |
| 998 | XD-1095 | Create a shell command for stopping all job executions | NULL | 2 |
| 999 | XD-1096 | Make Batch Job controllers HATEOAS compliant | Currently, the BatchJobsController and BatchJobExecutionsController are not HATEOAS compliant and we need make them so. | 3 |
| 1000 | XD-1097 | Redo Hadoop distribution dependency management | The way we now include various Hadoop distributions is cumbersome to maintain. Need a better way of managing and isolating these dependencies on a module level rather than container level. | 8 |
| 1001 | XD-1098 | Integration tests for XD Installer | NULL | 8 |
| 1002 | XD-1099 | Controllers - Disallow negative pageSize values | Currently you can specify negative pageSize values - The controllers should validatate that. Right now an internal exception is being return, leaking internal details to the caller - E.g.: **http://localhost:9393/batch/executions?pageSize=-1** results in: {code} <errors> <error logref="BadSqlGrammarException"> <message> StatementCallback; bad SQL grammar [SELECT TOP -1 E.JOB_EXECUTION_ID, E.START_TIME, E.END_TIME, E.STATUS, E.EXIT_CODE, E.EXIT_MESSAGE, E.CREATE_TIME, E.LAST_UPDATED, E.VERSION, I.JOB_INSTANCE_ID, I.JOB_NAME FROM BATCH_JOB_EXECUTION E, BATCH_JOB_INSTANCE I WHERE E.JOB_INSTANCE_ID=I.JOB_INSTANCE_ID ORDER BY E.JOB_EXECUTION_ID DESC]; nested exception is java.sql.SQLSyntaxErrorException: user lacks privilege or object not found: TOP </message> </error> </errors> {code} | 2 |
| 1003 | XD-1100 | Create aggregator module that uses redis as a message store | There is a need for a persistent store for messages so that they can survive the crash of a container node. The redis implementation in SI is useful since users may already be using redis for other features such as analytics. This module would sit alongside the current in-memory based aggregator. It brings in redis as a dependency and there should be configuration options exposed so that one can use a different version of redis as compared to the one that maybe used for analytics. Due to the need to have redis on the CP, and the large number of different options that each message store implementation provides and the different 3rd party library dependencies, I would like to avoid going down the path of using a profile here as it would seem to go beyond what we had discussed for profile support. We can revisit as the module contribution story based on boot unfolds. | 4 |
| 1004 | XD-1101 | Create aggregator module that uses an embedded database stored in the local filesystem | Similar to XD-1100. SI has the jdbc based message store. <int-jdbc:message-store id="messageStore" data-source="dataSource" table-prefix="MY_INT_"/> The configuration of this aggregator would be configured so that it uses an embedded database, hsqldb or H2 depending if there is any real perf benefit to one or the other, and store the data on the local file system. | 8 |
| 1005 | XD-1102 | Create microbenchmark for performace of redis and jdbc based aggregators | would be good to have a general feel for the general performance of these two options. Redis can run on the same node as the benchmark. | 8 |
| 1006 | XD-1103 | JDBC sink is broken - looks like some config options got booted | The JDBC sink is broken. Simple "time | jdbc" results in: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [insert into test (payload) values(?)]; nested exception is java.sql.SQLSyntaxErrorException: user lacks privilege or object not found: TEST Looks like some config options got clobbered during bootification. | 5 |
| 1007 | XD-1104 | Create Shell Integration test fixture for jdbc related sink | Would be nice to have some kind of regression testing on the jdbc sink, as it becomes more prominent in XD. Use of an in memory db where we expose eg a JdbcTemplate to assert state | 5 |
| 1008 | XD-1105 | Add some test coverage to mqtt modules | Even though it may be hard to come up with a mqtt broker, an easy test that should be automated is somesource | mqtt --topic=foo with mqtt --topics=foo | somesink And asserting that what is emitted to somesource ends up in somesink. | 3 |
| 1009 | XD-1106 | Create OOTB batch job that moved data from JDBC to HDFS | Basic inverse of current hdfsjdbc job. | 6 |
| 1010 | XD-1107 | Create Mock on DistributedJobService instead of instantiating object | Currently, the tests use instantiated DistributedJobService object instead of a simple mock. We can just use mock object for the tests. | 1 |
| 1011 | XD-1108 | Restore lax command line options | Restore --foo=bar as well as --foo bar Validation of values should be done as a separate story | 2 |
| 1012 | XD-1109 | Provide cmdline options validation | Wherever they come from (cmd line args or ENV_VARS), options such as transport, analytics, etc should be validated and issues should be reported to users | 2 |
| 1013 | XD-1110 | Fix JobRepoTests to use different batch job repo | Currently, the JobRepoTests use the same batch job repository that the XD runtime uses. Since the batch job repo doesn't delete the job instances, there would be stale data from this test. | 2 |
| 1014 | XD-1111 | Clear Redis after tests | The following keys remain after running the test suite: {code} redis 127.0.0.1:6379> keys * 1) "containers" 2) "containers.application:9292" {code} | 2 |
| 1015 | XD-1113 | ApplicationContext ID generation | ApplicationContext ID generation is difficult in general. Cloud Foundry solves this problem for us by providing unique instance ids to all running instances of an app. I don't suppose that helps much in the general case though, and we need something (a rule of thumb) that is unique and preferably deterministic, so that nodes retain their ID across process and connector restarts. In Cloud Foundry the instance id plays a vital role (and will be automatically picked up by the app and applied - need to look at how that plays in a context hierarchy). User can set the context id manually using {{spring.application.name}} and {{spring.application.index}}. | 2 |
| 1016 | XD-1114 | Investigate dropped Module Deployment Requests | We have observed in unit tests (see AbstractSingleNodeStreamIntegrationTests) that(Redis/SingleNode) occasionally fail. The root cause must be investigated further but there is some evidence to suggest that the control messages (ModuleDeploymentRequests) are not always received and handled by the ModuleDeployer. This does not produce an error but results in runtime stream failures. This problem may be resolved as part of the planned Deployment SPI but is being tracked here until we are certain that it has been resolved. | 5 |
| 1017 | XD-1115 | We no longer validate the --hadoopDistro options in the xd scripts | We no longer validate the --hadoopDistro options in the xd scripts. Seem sthe classes doing this validation were removed for boot. We do this validation in the xd-shell script | 3 |
| 1018 | XD-1116 | Add documentation about message store to aggregator doco | NULL | 2 |
| 1019 | XD-1117 | Add bash based scripts of simple module create to src/main/scripts | NULL | 1 |
| 1020 | XD-1118 | Add starting of newly build XD server, running of smoketest bash script, and killing of XD server to CI test | NULL | 5 |
| 1021 | XD-1119 | Create scripts for batch DB creation | We need to create a shell script that calls the batch DB creation sql files for the JDBC option selected. | 3 |
| 1022 | XD-1120 | Standardize Date/Time/TimeZone handling | We should we centrally standardize on date/time formats so that we don't create inconsistencies, and follow ISO 8601 internally. Internally we should only work with UTC (or make that the default config option). Ultimately, whatever the user sees is just a formatting concern. | 8 |
| 1023 | XD-1121 | Better error messages following XD-1109 | 2 forces at heand here: * Spring binding/validation itself * jsr303 (for eg @NotNull) | 2 |
| 1024 | XD-1122 | Add jmxPort to list of coerced cmd line options | Following merge of XD-1109. See discussion at https://github.com/spring-projects/spring-xd/commit/eaf886eab3b2ef07da55575029ccabb2c8a36af9#commitcomment-4701947 | 2 |
| 1025 | XD-1123 | Add profile activation on top of XD-953 | NULL | 5 |
| 1026 | XD-1124 | Turn off RestTemplate logging in shell | Example: {noformat} xd:>module delete --name sink:file 12:31:19,495 WARN Spring Shell client.RestTemplate:566 - DELETE request for "http://localhost:9393/modules/sink/file" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.xd.rest.client.impl.SpringXDException: Cannot delete non-composed module sink:file {noformat} The WARN log is redundant with the command result and should be silenced | 1 |
| 1027 | XD-1125 | Provide ModuleOptionsMetadata using simple approach | Following merge of XD-953, provide module options using the "simple" approach where applicable | 5 |
| 1028 | XD-1126 | Switch CAPITAL_LETTERS to system.property style in application config | NULL | 1 |
| 1029 | XD-1127 | Add support for deploying a batch job with partitioning across multiple XD nodes. | This is a very big story, needs some planning/discussion before starting work. Should be able to be implemented as a plugin. | 20 |
| 1030 | XD-1128 | Disable JMX by default (and extend to other contexts than modules) | NULL | 1 |
| 1031 | XD-1129 | Enable --refresh-dependencies into be present when executing gradlew | This could be inside gradlew or a .settings file. | 2 |
| 1032 | XD-1130 | Update the EC2 deployer | EC2 Deployer Needs to change its configuration behavior to use environment variables vs. the property files * Remove ConfigureSystem class, since we will use environment variables instead * Allow users to set environment variables via the xd-ec2 properties. * If properties are not present use those that are available in the application.yml * Utilize JClouds environment variable setup features to implement this behavior. | 5 |
| 1033 | XD-1131 | Basic support for Plugin contributed Module Options Metadata | Pending a better approach and extension point (see XD-1050), provide a way to factor out common recurring options for modules: * inputType * outputType * job parameters | 4 |
| 1034 | XD-1132 | JMS Module - add support for TOPICS | As a Spring XD user I need to listen on a JMS Topic and ingest the messages, so I can process the messages. Currently the module only allows for Queues | 2 |
| 1035 | XD-1133 | Create test script that reproduces failure in writing to HDFS when undeploying and redeploying a stream | See https://github.com/spring-projects/spring-xd/pull/415#issuecomment-29024329 This test should be created so that it reproduces the failure and then turned off, so that when we switch to the new HDFS writing code | 2 |
| 1036 | XD-1134 | Logging is not producing a log file | * need to add a -Dxd.home=$XD_HOME to the start scripts else all files will write to the /logs directory. * Need to update .bat files to use % for env variables instead of $. * Need to rename logger.config to logging.config so that boot will pick up the log config files. * Admin needs to use xd-admin-logger configs instead of xd-container-logger * Renamed logging file for singlenode from admin.log to singlenode.log | 1 |
| 1037 | XD-1135 | Local control channels enabled in SingleNodeApplication when alternate transport is selected. | This causes intermittent test failures when testing streams with other transports since ModuleDeployer receives duplicate requests from multiple threads. Fix is to check for local transport before invoking setUpControlChannels() | 1 |
| 1038 | XD-1136 | Failure in writing to HDFS when undeploying and redeploying a stream with numbers in directory and/or file name | NULL | 3 |
| 1039 | XD-1137 | Have the REST info about a module advertise enum properties | When a ModuleOption is backed by an enum, change the (currently String) type representation to be the possible values ie java.lang.String -> String but traffic.Light -> Red | Green | Orange | 3 |
| 1040 | XD-1138 | Review and Optimize the Serialized JSON Nodes for Batch Objects | For several Batch Job related JSON endpoints, we serialize too much information. | 4 |
| 1041 | XD-1139 | Add TaskLet to Stream from (S)FTP to HDFS | NULL | 3 |
| 1042 | XD-1140 | Add (S)FTP Gateway and Batch Partitioner to List/Process Remote Files | NULL | 16 |
| 1043 | XD-1142 | Spike to model the cluster nodes. | Each node in the cluster advertises itself and the admin node listens to these ads and creates groups out of them. The deployment of jobs and stream processing can then be deployed onto a specific group and specific nodes within the group. The project https://github.com/spring-projects/spring-data-grid is the start of this model. | 8 |
| 1044 | XD-1143 | Spike for matching algorithm for stream/job deployments and nodes | Base matching algorithm on the model used in http://research.cs.wisc.edu/htcondor/ Jobs/Stream - specify their requirements and preferences Nodes - specify their requirements and preferences e.g. A job/stream module "requires" a linux x85-64 platform and "prefers" the machine with the most free memory A node "requires" that only only can run jobs when there is 25% or more free memory and it "prefers" to run stream modules over job modules. The requirements and preferences are represented as SpEL expressions in a "Advertisement" data structure. There is a 'require' expression and a 'preference' expression. Matching occurs between Node Ads and Job/Stream Ads so that the requirements of both Ads evaluate to true and the matching nodes are ranked according to the preference expression. The Job/Streams Ads can make use of well defined attributes about the nodes, such as it's memory/cpu usage. | 10 |
| 1045 | XD-1144 | Spike for transformation of the parsed stream/job definition to a Physical Deployment Model taking into account a Deployment Manifest | The initial parsing of a stream/job definitions into a list of ModuleDeploymentRequests needs to be transformed into a Physical Deployment Model that takes into account 1) module co-location 2) partitioning 3) number of instances 4) node assignment an potentially other data related to the runtime properties of a module (e.g. concurrency settings) The an external DeploymentManifest will be used to capture this information. A deployment of a stream or job will then need to optionally provide a reference to deployment manifest. | 10 |
| 1046 | XD-1145 | Add --date option to the trigger module | The TriggerSourceOptionsMetadata class should be able to use an actual Date object, thanks to Spring binding conversion. BUT, the ${date} construct will receive a toString version of it. Make sure this works properly | 4 |
| 1047 | XD-1146 | With Partitioned Jobs, Wire Partitioner and StepExecutionHandlers with the MessageBus | NULL | 16 |
| 1048 | XD-1148 | Allow local data transport option for the container | Currently local is not a supported data transport for the container. It should be an option. Note that local is not valid for control transport for a standalone container. So we need to revisit the current semantics that default the control transport to be the same as the data transport, i.e. ,--transport=local should fail. | 2 |
| 1049 | XD-1149 | source jms --- connect to another jms provider | As a system administrator I need to connect to SonicMQ as jms provider When setting up the correct spring xml file and added the correct jar files to the lib directory I received the following exception--- Question: is there a spot I should be defining the conversion strategy? {code} . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v0.5.0.M6) 15:04:36,092 ERROR http-bio-9393-exec-1 rest.RestControllerAdvice:157 - Caught exception while handling a request org.springframework.integration.MessageHandlingException: error occurred in message handler [moduleDeployer] org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:79) org.springframework.integration.config.ServiceActivatorFactoryBean$1.handleRequestMessage(ServiceActivatorFactoryBean.java:83) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:142) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:94) org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:42) org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:86) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:228) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:212) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:177) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:171) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:149) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.xd.dirt.stream.DeploymentMessageSender.sendDeploymentRequests(DeploymentMessageSender.java:57) org.springframework.xd.dirt.stream.AbstractDeployer.sendDeploymentRequests(AbstractDeployer.java:137) org.springframework.xd.dirt.stream.AbstractDeployer.basicDeploy(AbstractDeployer.java:157) org.springframework.xd.dirt.stream.AbstractInstancePersistingDeployer.deploy(AbstractInstancePersistingDeployer.java:78) org.springframework.xd.dirt.rest.XDController.save(XDController.java:242) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:214) org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:748) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:947) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:878) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:946) org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:848) javax.servlet.http.HttpServlet.service(HttpServlet.java:647) org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:822) javax.servlet.http.HttpServlet.service(HttpServlet.java:728) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:88) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:744) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in file [/Users/dmarley/sandbox/spring-xd/build/dist/spring-xd/xd/modules/source/jms/config/../../../common/jms-sonic-infrastructure-context.xml]: Initialization of bean failed; nested exception is org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'progress.message.jclient.ConnectionFactory' to required type 'javax.jms.ConnectionFactory' for property 'targetConnectionFactory'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [progress.message.jclient.ConnectionFactory] to required type [javax.jms.ConnectionFactory] for property 'targetConnectionFactory': no matching editors or conversion strategy found org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:547) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:300) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:296) org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195) org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:660) org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760) org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) org.springframework.boot.SpringApplication.refresh(SpringApplication.java:552) org.springframework.boot.SpringApplication.run(SpringApplication.java:293) org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:130) org.springframework.xd.module.SimpleModule.initialize(SimpleModule.java:135) org.springframework.xd.dirt.module.ModuleDeployer.deploy(ModuleDeployer.java:239) org.springframework.xd.dirt.module.ModuleDeployer.deployModule(ModuleDeployer.java:229) org.springframework.xd.dirt.module.ModuleDeployer.handleDeploy(ModuleDeployer.java:214) org.springframework.xd.dirt.module.ModuleDeployer.handleDeploymentRequest(ModuleDeployer.java:196) org.springframework.xd.dirt.module.ModuleDeployer.handleMessageInternal(ModuleDeployer.java:137) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) ... 63 more Caused by: org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'progress.message.jclient.ConnectionFactory' to required type 'javax.jms.ConnectionFactory' for property 'targetConnectionFactory'; nested exception is java.lang.IllegalStateException: Cannot convert value of type [progress.message.jclient.ConnectionFactory] to required type [javax.jms.ConnectionFactory] for property 'targetConnectionFactory': no matching editors or conversion strategy found org.springframework.beans.BeanWrapperImpl.convertIfNecessary(BeanWrapperImpl.java:474) org.springframework.beans.BeanWrapperImpl.convertForProperty(BeanWrapperImpl.java:505) org.springframework.beans.BeanWrapperImpl.convertForProperty(BeanWrapperImpl.java:499) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.convertForProperty(AbstractAutowireCapableBeanFactory.java:1497) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1456) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1192) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537) ... 81 more Caused by: java.lang.IllegalStateException: Cannot convert value of type [progress.message.jclient.ConnectionFactory] to required type [javax.jms.ConnectionFactory] for property 'targetConnectionFactory': no matching editors or conversion strategy found org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:267) org.springframework.beans.BeanWrapperImpl.convertIfNecessary(BeanWrapperImpl.java:459) ... 87 more {code} | 4 |
| 1050 | XD-1150 | Update docs for separate control and data transport | NULL | 1 |
| 1051 | XD-1151 | Apply Composite GoF pattern to ModuleDefinition | NULL | 8 |
| 1052 | XD-1152 | Step execution progress Shell command to use coherent Id (jobExecutionId:stepExecutionId) | Instead of using jobExecutionId and stepExecutionId as two separate options for the "job execution step progress" command, we can have a single option with id mentioned as (jobExecutionId:stepExecutionId) | 2 |
| 1053 | XD-1153 | Fix Tail Source to Use Native Adapter by Default | Support all available options | 3 |
| 1054 | XD-1154 | Update doc about trigger changes | Trigger has been changed to be one single modules, taking params (as opposed to 3 before) Update doc at https://github.com/spring-projects/spring-xd/wiki/Batch-Jobs Also, arguably, the module could be advertised as a source itself (it is only mentioned in the context of batch) | 3 |
| 1055 | XD-1155 | The lib directory for hadoop12 contains mix of hadoop versions | This causes issues depending on which version of the core/common jar gets loaded first - like: xd:>hadoop fs ls -ls: Fatal internal error java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:213) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2401) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2411) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190) at org.apache.hadoop.fs.shell.Command.run(Command.java:154) at org.apache.hadoop.fs.FsShell.run(FsShell.java:255) at org.springframework.xd.shell.hadoop.FsShellCommands.run(FsShellCommands.java:412) at org.springframework.xd.shell.hadoop.FsShellCommands.runCommand(FsShellCommands.java:407) at org.springframework.xd.shell.hadoop.FsShellCommands.ls(FsShellCommands.java:110) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:191) at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:64) at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:48) at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:127) at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:483) at org.springframework.shell.core.JLineShell.run(JLineShell.java:157) at java.lang.Thread.run(Thread.java:724) | 3 |
| 1056 | XD-1156 | Refactor/Simplify JobPlugin | Currently, the JobPlugin extends AbstractPlugin and the AbstractPlugin has got lots of unused code (like it doesn't do anything with preProcessSharedContext()) in there. | 2 |
| 1057 | XD-1157 | Update docs for gemfire sink to include locator configuration | NULL | 1 |
| 1058 | XD-1158 | Create integration test script for JMS & MQTT | Create a script to sanity check JMS and MQTT | 3 |
| 1059 | XD-1159 | Add a MongoDB Sink | This should be quite straightforward, since the Spring Data Mongo jars are already included. We have this working by just adding the attached sink context file and the spring-integration-mongodb jar. (This works for JSON string streams, but a mongo converter probably needs added to support Tuple conversion) | 5 |
| 1060 | XD-1160 | Standardize naming and unit for options across modules | We should standardize on the options between modules: idleTimeout - timeout rolloverSize - rollover Also, need to standardize on unit used for timeout - should this be s or ms? | 8 |
| 1061 | XD-1161 | Re-deployment of hdfs sink reuses filename of first deployment | Need to check for existing files with the same file counter | 3 |
| 1062 | XD-1162 | Column option of JDBC sink should not convert underscore to property name. | Current implementation of column option of JDBC sink convert underscore to java property name. If database column contains underscore, there is no way to store data. So JdbcMessagePayloadTransformer should not use JdbcUtils.convertUnderscoreNameToPropertyName even if column contains "_". | 1 |
| 1063 | XD-1163 | Update Spring Framework dependency to 4.0 GA | NULL | 1 |
| 1064 | XD-1164 | Update Spring Integration version to 4.0.M2 | NULL | 1 |
| 1065 | XD-1165 | Enhancements to Gemfire CQ Source | The Gemfire CQ source needs some enhancements: * enable locator configuration * consider decoupling from JSON. Currently designed to work with gemfire-json-server to avoid dependence on specific domain objects on the client and server side. So produces json strings from PdxInstance(s) stored in the cache. | 2 |
| 1066 | XD-1166 | Create Gemfire Integration Test Scripts | NULL | 3 |
| 1067 | XD-1167 | Mail Source ModuleOptions (+ profiles) | NULL | 2 |
| 1068 | XD-1168 | Tapping a stream with multiple labelled filters causes duplicate messages | Test case is here: https://github.com/spring-projects/spring-xd/blob/master/src/test/scripts/basic_stream_tests#L49 We would expect one message in the counter, but get 3. | 5 |
| 1069 | XD-1169 | Column name of JDBC sink module should not hard code to "payload". | Current implementation of JDBC sink module insert data into "payload" column. But I can't just change the default column "payload" to something else using --columns option. Because JdbcMessagePayloadTransformer compare the columnName to "payload" and it hard coded. | 1 |
| 1070 | XD-1170 | Splunk module is broken | Splunk sink module doesn't work at all. It throws java.lang.VerifyError exception like following. nested exception is java.lang.VerifyError: class org.springframework.integration.splunk.outbound.SplunkOutboundChannelAdapter overrides final method onInit.()V This is because SplunkOutputChannelAdapter refers old spring integration jar, but recent AbstractReplyProducingMessageHandler (which SplunkOutputChannelAdapter extends) set final to onInit method. Hence it doesn't work. SplunkOutboundChannelAdapter should be fixed to not override onInit method and replace the jar file spring-integration-splunk-1.0.0.M1.jar. | 2 |
| 1071 | XD-1171 | Container (Launcher) id is not unique | XD container's id is set to use its application context id which is derived from: ${vcap.application.name:${spring.application.name:${spring.config.name:application}}}:${vcap.application.instance_index:${spring.application.index:${server.port:${PORT:null}}}} With the *default* values[the PORT is not set and embedded tomcat uses local port], the launcher id is set to "application:0" When I have multiple launchers then all the launchers have the same id as "application:0" which doesn't seem correct. Do we need to use the Id that is generated at the XDContainer's constructor here? public XDContainer() { this.id = UUID.randomUUID().toString(); } | 3 |
| 1072 | XD-1172 | Restore previous CmdLine library to populate options | Expected benefits are --key<space>value as well as --key=value on the command line (eg XD-1108) nice usage screen | 4 |
| 1073 | XD-1173 | Duplicate messages on tap | If I fiddle with the testTappingWithLabels method I can reproduce the same issue: HttpSource source = newHttpSource(); FileSink sink = newFileSink().binary(true); FileSink tapsink1 = newFileSink().binary(true); stream().create("myhttp", "%s | flibble: transform --expression=payload.toUpperCase() | flibble2: transform --expression=payload.toUpperCase() | %s", source, sink); stream().create("mytap4", "tap:stream:myhttp.flibble > transform --expression=payload.replaceAll('A','.') | %s", tapsink1); source.ensureReady().postData("Dracarys!"); assertThat(sink, eventually(hasContentsThat(equalTo("DRACARYS!")))); assertThat(tapsink1, eventually(hasContentsThat(equalTo("DR.C.RYS!")))); java.lang.AssertionError: Expected: "DR.C.RYS!", trying at most 10 times but: failed after 10*100=1000ms: "DR.C.RYS!DR.C.RYS!", | 3 |
| 1074 | XD-1174 | Update HDFS sink documentation to reflect new functionality introduced in XD-990 and XD-991 | NULL | 2 |
| 1075 | XD-1175 | Support external defaults for PojoModuleOptions | The current behavior when there are "global" external defaults to module options is to set them in the placeholder construct: ${foo:${the_default}} where ${the_default} is sourced from some properties file. The downside is that the module options infrastructure is unaware of them. Provide support for such defaults (at least when using PojoModuleOptions) in the form of * Annotate a field (or the setter?) with @Value("${the_default}") * (maybe) Annotate the pojo with @PropertySource to indicate the location of the properties file * (maybe) come up with a general naming scheme for the properties file | 5 |
| 1076 | XD-1176 | Update to spring-data-hadoop 2.0.0.M4 | Update dependencies to spring-data-hadoop 2.0.0.M4 | 1 |
| 1077 | XD-1177 | Change module naming strategy in parser taking into account no duplicate labels and repeated modules | Labels should only be used help uniquely identify a module in a stream. The stream definition a | x: b | x: c | d should throw an error since the label x is applied to two modules. Also ambiguity exists with a stream definition such as http | transform | transform | filter | hdfs since a module with the name 'transform' will exist twice, meaning a tap on transform will result in message from both transform modules. To make the naming unique and the usage with taps, the following naming strategy is proposed demonstrated by example. http | transform | transform | filter | hdfs http, transform.0, transform.1, filter, hdfs If a use tries to tap on transform, an error would be shown saying that you could try to tap on transform.0 or transform.1 or alternatively use labels. http | transform | filter | transform | hdfs http, transform.0, filter, transform.1, hdfs http | x: transform | filter | transform | hdfs http, x, filter, transform, hdfs | 4 |
| 1078 | XD-1178 | Make avro sink options consistent with hdfs and add docs | NULL | 3 |
| 1079 | XD-1179 | Nodes can not connect with Admin using Redis as transport | This has happened more than once, where a node fails for whatever reason, and when it is restarted, it does not receive requests from the admin server. This could be file handle count based. Since this is not Rabbit as a transport I'm not chasing this down yet. But felt it needed to be recorded. | 5 |
| 1080 | XD-1180 | Enabling of JMX support is broken | However this is triggered (depending on whether https://github.com/spring-projects/spring-xd/pull/477/files is merged yet or not), jmx seems to be broken because of duplicate beans / mbeans names | 4 |
| 1081 | XD-1181 | Surface the provenance of a default to the user | When using @Value for providing a default value in a module options POJO, make it so that the REST API (and hence the module info command) advertises that 1) the expression was ${foo.something} 2) to the best extent possible (value may come from another property source), tell which config file it came from (introspecting the @PropertySource annotation) | 3 |
| 1082 | XD-1182 | Update to spring-data-hadoop 2.0.0.M5 | Update to spring-data-hadoop 2.0.0.M5 when it is released and remove the temporary DatasetTemplateAllowingNulls in spring-xd-hadoop We should also review the supported hadoop distros - think we should support anything that is current/stable: - hadoop12 - hadoop22 - phd1 (PHD 1.1) - hdp13 - hdp20 - cdh4 | 3 |
| 1083 | XD-1183 | topic in mqtt source was marked as topics | Changed field back to topic | 1 |
| 1084 | XD-1184 | Admin & Launcher startup fails when XD_JMX_ENABLED is set to true | When exporting of MBeans are enabled via XD_JMX_ENABLED (also, jmxEnabled as in application.yml), the Admin and Lancher server application fail to start. Since the admin applications has the same 'integrationMbeanExporter' bean name for IntegrationMBeanExporter as that its ParentConfiguration, there is a naming conflicts and the exception thrown as: (Same is the case for launcher and its parent configuration) Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mbeanExporter' defined in class org.springframework.context.annotation.MBeanExportConfiguration: Invocation of init method failed; nested exception is org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [org.springframework.integration.monitor.IntegrationMBeanExporter@1de40d3e] with key 'integrationMBeanExporter'; nested exception is javax.management.InstanceAlreadyExistsException: org.springframework.integration.monitor:name=integrationMBeanExporter,type=IntegrationMBeanExporter org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304) org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300) org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195) org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:700) org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760) org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:124) org.springframework.boot.SpringApplication.refresh(SpringApplication.java:609) org.springframework.boot.SpringApplication.run(SpringApplication.java:321) org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:130) org.springframework.xd.dirt.server.AdminServerApplication.run(AdminServerApplication.java:60) org.springframework.xd.dirt.server.AdminServerApplication.main(AdminServerApplication.java:42) Caused by: org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [org.springframework.integration.monitor.IntegrationMBeanExporter@1de40d3e] with key 'integrationMBeanExporter'; nested exception is javax.management.InstanceAlreadyExistsException: org.springframework.integration.monitor:name=integrationMBeanExporter,type=IntegrationMBeanExporter org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstance(MBeanExporter.java:610) org.springframework.jmx.export.MBeanExporter.registerBeans(MBeanExporter.java:535) org.springframework.jmx.export.MBeanExporter.afterPropertiesSet(MBeanExporter.java:417) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1612) org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549) ... 15 more Caused by: javax.management.InstanceAlreadyExistsException: org.springframework.integration.monitor:name=integrationMBeanExporter,type=IntegrationMBeanExporter com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513) org.springframework.jmx.support.MBeanRegistrationSupport.doRegister(MBeanRegistrationSupport.java:195) org.springframework.jmx.export.MBeanExporter.registerBeanInstance(MBeanExporter.java:663) org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstance(MBeanExporter.java:600) ... 19 more | 2 |
| 1085 | XD-1185 | Add redisConnectionFactory with connection pool | We need to add a connection pool to the Redis connection factory used for the transport, otherwise we'll see exceptions like these: 12:57:54,842 ERROR inbound.tictoc.0-redis:queue-inbound-channel-adapter1 inbound.RedisQueueMessageDrivenEndpoint:183 - Failed to execute listening task. Will attempt to resubmit in 5000 milliseconds. org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is com.lambdaworks.redis.RedisException: Unable to connect at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:46) at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:36) at org.springframework.data.redis.connection.lettuce.LettuceConverters.toDataAccessException(LettuceConverters.java:159) at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:253) at org.springframework.data.redis.connection.lettuce.LettuceConnection.bRPop(LettuceConnection.java:1508) at org.springframework.data.redis.core.DefaultListOperations$12.inRedis(DefaultListOperations.java:163) at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:51) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:185) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153) at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86) at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:160) at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:105) at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:178) at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:51) at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:291) at java.lang.Thread.run(Thread.java:724) Caused by: com.lambdaworks.redis.RedisException: Unable to connect at com.lambdaworks.redis.RedisClient.connect(RedisClient.java:176) at com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:139) at org.springframework.data.redis.connection.lettuce.LettuceConnection.getAsyncDedicatedConnection(LettuceConnection.java:2924) at org.springframework.data.redis.connection.lettuce.LettuceConnection.getDedicatedConnection(LettuceConnection.java:2932) at org.springframework.data.redis.connection.lettuce.LettuceConnection.bRPop(LettuceConnection.java:1506) ... 11 more Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70) at org.jboss.netty.channel.Channels.connect(Channels.java:634) at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:207) at com.lambdaworks.redis.RedisClient.connect(RedisClient.java:165) ... 15 more | 1 |
| 1086 | XD-1186 | Support use of application.yml fragments | Since spring boot (by default) looks for property sources from file:./config/application.yml and file:./application.yml In XD bundles, the CLASSPATH of our XD startup scripts'(admin, container, singlenode startup scripts) set to use APP_HOME and APP_HOME/config. But, if we have an application.yml fragment on $APP_HOME/config, then it is considered as classpath resource and the actual "application.yml" (from dirt lib/ classpath) is not loaded. Also, we need to separate out the properties files we have inside $APP_HOME/config as by default, boot uses "config" directory as well. | 3 |
| 1087 | XD-1187 | xd-admin server to --transport as an option. | --transport as an alias for --controlTransport | 3 |
| 1088 | XD-1188 | Add ModuleOptions support for Rabbit sink | We need to add ModuleOptions support for Rabbit sink. | 2 |
| 1089 | XD-1189 | Could not override rabbit sink module's rabbit connection factory properties | Currently RabbitMQ sink module's connection properties could not be overridden by "${xd.config.home}/${configProperties:rabbit}.properties" even if "local-override" is set to true. It looks like the AmqpTemplate used by the AMQP outbound channel adapter doesn't use the connection factory defined in the sink module. | 2 |
| 1090 | XD-1190 | Setup precedence order for module properties' property resolver | The PropertyResolver needs to follow the below precedence order on PropertySources when resolving the module properties: From lowest to the highest order, 0 application.yml 1 applicaiton.yml fragment 2 property placeholders 2a property placeholder under 'shared' config directory 2b property placeholder under module/(source/sink/processor)/config directory 3. environment variables 4. system properties 5. command line | 5 |
| 1091 | XD-1191 | JDBC sink destroys existing table | The jdbc sink deletes existing table and creates a single column payload one even if properties file has 'initializeDatabase=false' | 3 |
| 1092 | XD-1192 | Add documentation for JDBC to HDFS batch job | Add docs to section https://github.com/spring-projects/spring-xd/wiki/Batch-Jobs#pre-packaged-batch-jobs | 2 |
| 1093 | XD-1193 | Update to Spring Batch 2.2.4 | NULL | 1 |
| 1094 | XD-1194 | CommandLine default values from container & admin options can not be overridden | If we have default values from Container/Admin options, then they can not be overridden by the system properties or system environment. Currently, the only default we have for the Container/Admin options is for "jmxEnabled" option and since it is a boolean it can never be overridden by sys/env property XD_JMX_ENABLED. I think we need to make sure there are no default values assigned for the non-boolean Container/Admin options and handle the boolean type option separately. | 2 |
| 1095 | XD-1195 | Add paging and sorting to Field Value Counter API | see discussion at https://github.com/spring-projects/spring-xd/commit/2f0e80b5e337b71c9c70de510a44d2f050d10fa7 | 5 |
| 1096 | XD-1196 | Align filehdfs batch job defaults with those of corresponding hdfs sink | The batch jobs use different defaults compared to some of the sink/source modules. filehdfs puts data in a /data directory with files named after the stream using a .log file extension. The hdfs sink puts files in an /xd/<streamname> directory using .txt as the default file extension. filehdfs needs a more descriptive naming | 5 |
| 1097 | XD-1197 | job launch doesn't do tab completion of --name | In the shell: job launch doesn't do completion of --name, this is different behavior compared to job destroy typing 'job destroy' and hitting tab completes with '--name' typing 'job launch' and hitting tab does nothing | 5 |
| 1098 | XD-1198 | Shell: Job Parameters are whitespace sensitive | This works: {code} job launch helloSpringXD --params "{"myStringParameter":"foobar","secondParam":"hello"}" {code} This fails: {code} job launch helloSpringXD --params "{"myStringParameter":"foobar", "secondParam":"hello"}" {code} Notice the space between the parameters. This fails with: {code} spring> job launch helloSpringXD --params "{"myStringParameter":"foobar", "secondParam":"hello"}" 14/01/06 12:00:07 WARN client.RestTemplate: PUT request for "http://localhost:9393/jobs/helloSpringXD/launch" resulted in 500 (Internal Server Error); invoking error handler Command failed org.springframework.xd.rest.client.impl.SpringXDException: org.springframework.integration.MessageHandlingException: java.lang.IllegalArgumentException: Unable to convert provided JSON to Map<String, Object> {code} Internally the parameters are parsed to: {code} {"myStringParameter":"foobar",secondParam":"hello"} {code} The curly brace is missing. | 4 |
| 1099 | XD-1199 | Fix filejdbc batch job | filejdbc throws an exception: {code} java.lang.IllegalArgumentException: Could not resolve resource location pattern [/mycsvdir/*.csv]: class path resource [mycsvdir/] cannot be resolved to URL because it does not exist {code} This can be solved by using a file:// prefix Maybe just update the docs? | 3 |
| 1100 | XD-1200 | Fix hdfsjdbc batch job | hdfsjdbc throws an exception: {code} org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'itemReader' defined in URL [file:/Users/trisberg/Projects/spring-xd/build/dist/spring-xd/xd/modules/job/hdfsjdbc/config/hdfsjdbc.xml]: Could not resolve placeholder 'columns' in string value "${columns}" {code} The hdfsjdbc job uses 'columns' instead of 'names' as the parameter for the column-names. Should we make this usage consistent between jobs? There is a comment in the docs - "there is also a limitation in that the database table must be created manually. This is due to a bug in Spring Hadoop and will be fixed in the future." Think this is this solved in Spring Hadoop now? initializeDatabase should default to false now to be consistent with jdbc sink Rename batch-jdbc/mongo-import.properties to batch-jdbc/mongo.properties since these aren't just for import | 5 |
| 1101 | XD-1201 | Create custom help command for xd-shell so that hadoopDistro option is listed | xd-shell is using the default spring-shell help command. Need to create a help command specific to XD so that it can list the --hadoopDistro command line option. Note, the hadoopDistro command line option is actually processed by the xd-shell bash script | 2 |
| 1102 | XD-1202 | Update build script to use correct version of spring-data-hadoop based on distro | NULL | 1 |
| 1103 | XD-1203 | JobPlugin - Use containsKey when checking for Parameters | Use containsKey when checking for Parameters Add tests | 2 |
| 1104 | XD-1204 | Update jdbchdfs properties and defaults to better match hdfs sink | We should use fileName, fileExtension properties and default to /xd/jobname as directory | 3 |
| 1105 | XD-1205 | Add test for filejdbc to test scripts | The filejdbc jobs isn't included in the test scripts | 3 |
| 1106 | XD-1206 | Garbage in Job Repo Causes List Failure | For some reason, the partitioned batch jobs are storing {{StepExecutions}}s in the DB with a null ID. This causes {{JobCommandTests.testListStepExecutionsForSpecificJobExecution()}} to fail because the HATEOS code asserts not null... {code} 18:25:26,185 ERROR http-nio-9393-exec-3 rest.RestControllerAdvice:186 - Caught exception while handling a request java.lang.IllegalArgumentException: [Assertion failed] - this argument is required; it must not be null org.springframework.util.Assert.notNull(Assert.java:112) org.springframework.util.Assert.notNull(Assert.java:123) org.springframework.hateoas.mvc.ResourceAssemblerSupport.createResourceWithId(ResourceAssemblerSupport.java:87) org.springframework.xd.dirt.rest.StepExecutionInfoResourceAssembler.toResource(StepExecutionInfoResourceAssembler.java:39) org.springframework.xd.dirt.rest.BatchStepExecutionsController.list(BatchStepExecutionsController.java:107) {code} For some reason, the query for objects with the {{jobExecutionId}} also returned these two objects with null keys. Blowing away my data directory fixes the problem (until I run another partitioned batch job). I need to figure out why spring batch is creating these bad records, but we should probably add some defensive code to protect against null IDs. You can reproduce by building against my XD-1146 branch. | 1 |
| 1107 | XD-1207 | Job Plugin - Notification Channel not correctly bound to MessageBus | NULL | 4 |
| 1108 | XD-1208 | Create Module options metadata for OOTB jobs | NULL | 5 |
| 1109 | XD-1209 | Add support for XD_CONFIG environment variable in windows shell scripts | This was added in bash scripts as part of XD-1186. | 4 |
| 1110 | XD-1210 | Shell - 'makeUnique' Job Parameter is true by default | Currently the shell assumes that the 'makeUnique'job parameter is by default *false*. That is not true. As a consequence the parameter has currently no impact/does not work. | 2 |
| 1111 | XD-1211 | Update hadoop instructions in the xd-samples | batchHashtagCount and batchWordCount projects need "hadoop fs ls" instructions need to be updated. | 1 |
| 1112 | XD-1212 | Boot updates post 0.5 M7 | Some boot classes we compile against have changed or been replaced. | 3 |
| 1113 | XD-1213 | Allow conditional validation for module options | Using JSR303 groups, which should be derived from injected values | 4 |
| 1114 | XD-1214 | Make job names unique across tests that use the same JobRepository | The job names used by the tests should be unique across the tests when use the same JobRepository. | 3 |
| 1115 | XD-1215 | Distributed JobLocator should return a valid job | In distributed batch processing in XD, the JobLocator implementation for getJob(String jobName) should return a valid Job (FlowJob/SimpleJob). Since we won't we relying on the MapJobRegistry's joblocator implementation which doesn't work in distributed use case, we need to have an appropriate way to return FlowJob/SimpleJob using XD's BatchJobLocator. | 5 |
| 1116 | XD-1216 | Serialization of ChunkContext fails using Kryo | NULL | 2 |
| 1117 | XD-1217 | twittersearch and twitterstream should support compatible formats | Currently twitterstream emits native twitter json whereas twittersearch uses SI/Spring Social and emits spring social Tweet types. This makes it difficult to replace twitter sources and reuse XD stream definitions. This requires coordination with SS 1.1.0 and SI 4.0 GA releases. NOTE: I think it's a good idea to continue to support native twitter JSON, keep as an option for twitterstream, but the default should be Tweet types. | 5 |
| 1118 | XD-1218 | Base server implementation on Spring Boot | Server startup is done using Spring Boot as well as starting a module application context at runtime. | 20 |
| 1119 | XD-1219 | Release 1.0.0.M5 | NULL | 8 |
| 1120 | XD-1220 | Batch jobs should use application.yml provided connection as default | Batch jobs should use application.yml provided connection as default. They now have their own configuration in batch-jdbc.properties. This config needs to account for any changes made to application.yml settings so the data is written to the batch metadata database by default. | 5 |
| 1121 | XD-1221 | XD modules should not have 'build' directory upon running gradle build | Currently, the 'modules' project is marked as java project to enable eclipse/idea metadata files generation. But it generates a 'build' directory with a jar that has empty MANIFEST file. This build directory also gets copied into the bundle after 'dist'. | 2 |
| 1122 | XD-1222 | Duplicate MBean server definition by MBeanExportingPlugin | The MBeanServer is referred by XD admin/launcher when JMX is enabled (by setting --jmxEnabled option) and this is defined in xd-global-beans.xml. The MBeans that are exposed by the modules also use the same MBeanServer define above and there is a duplicate MBeanServer definition (from jmx/common.xml) which the MBeanExportingPlugin adds as a component to the module which doesn't seem to be needed. Also, currently the flag "jmxEnabled" is generic and used by adminserver/launcher as well as the module MBeanExportingPlugin. If we have a separate flag to enable the JMX *only* for modules then, a separate definition of MBeanServer could be necessary. | 1 |
| 1123 | XD-1223 | Unit/Integration tests need appropriate cleanup of test data during teardown | Currently, there are test data(e.g: stream name) not being cleaned up during teardown especially when there is a test case failure. This breaks the test suite when the same test data is used by other tests. We need to move the cleanup logic at an appropriate level so that the test data is always cleaned up irrespective of the test result. | 5 |
| 1124 | XD-1224 | Add code coverage to gradle build | Build should be able to generate code coverage reports. After a quick tour of the intertubes it seems that JaCoCo is a well maintained project and has first class support inside gradle. http://www.gradle.org/docs/current/userguide/jacoco_plugin.html | 3 |
| 1125 | XD-1225 | Integrate code coverage reports into the CI process | Not sure if this is best done via Sonar our sonar build plan, the nightly one, or the frequent one off master... Open question is if we want to fail a build do to code coverage levels. | 4 |
| 1126 | XD-1226 | Review abstract base classes used in test cases to ensure proper resource cleanup. | NULL | 6 |
| 1127 | XD-1227 | Investigate long running tests and create refactoring issues | https://sonar.springsource.org/dashboard/index/7173?did=3 Shows which of our current tests take the most time to execute. xd.dirt.stream and xd.shell.command are where the most time is spent. In xd.dirt.stream it seems likely that time can be reduced by not restarting a new single node server per test, but sharing it across tests, e.g. RabbitSingleNodeStreamDeploymentIntegrationTests LocalSingleNodeStreamDeploymentIntegrationTests RedisSingleNodeStreamDeploymentIntegrationTests As a first pass, the test that take longer than 15 seconds in that report should be investigated. | 6 |
| 1128 | XD-1228 | Provide a easy, prescriptive means to perform unit and basic stream integration tests. | AbstractSingleNodeStreamDeploymentIntegrationTests is the basis of 'state of the art' testing for a stream that allows you to get a reference to the input and output channel of the stream http | filter | transform | file. One can send messages to the channel after the http module, but before filter and one can retrieve the messages that were sent to the channel after the transform module but before file. The current implementation inside AbstractSingleNodeStreamDeploymentIntegrationTests can be improved in terms of ease of use for end-users. The issue is to create as simple a way as possible for a user to test their processing modules/stream definitions without having to actually do a real integration test by sending data to the input module. Either as a separate issue or as part of this one, the documentation https://github.com/spring-projects/spring-xd/wiki/Creating-a-Processor-Module should be updated to explicitly show how to use this issue's test functionality. | 8 |
| 1129 | XD-1229 | Create build infrastructure for Angular based UI | Create a project that includes build and test automation for a new Angular based UI. This work is independent of calling the UI build step from gradle. A super minimal UI, just to have some basic code, is all that is required. This should use Grunt Jasmine Karma Bower YUIdoc (separate story?) UI Components from backbone should be available in the base project. | 8 |
| 1130 | XD-1230 | Integrate grunt based UI build into the XD's gradle build | Blog post http://naleid.com/blog/2013/01/24/calling-gruntjs-tasks-from-gradle/ seems to be the definitive reference.... | 5 |
| 1131 | XD-1231 | Investigate if we should use RequreJS with Angular | NULL | 6 |
| 1132 | XD-1232 | Create EC2 AMI for single-node install of Apache Hadoop 1.2.1 | NULL | 4 |
| 1133 | XD-1233 | Create EC2 AMI for single-node install of Apache Hadoop 2.2.0 | NULL | 3 |
| 1134 | XD-1234 | Create EC2 AMI for single-node install of Pivotal HD 1.1 | NULL | 8 |
| 1135 | XD-1235 | Create EC2 AMI for single-node install of Cloudera CDH 4.5.0 | NULL | 5 |
| 1136 | XD-1236 | Create EC2 AMI for single-node install of Hortonworks Data Platform 1.3 | NULL | 5 |
| 1137 | XD-1237 | Add option to stop all running XD EC2 instances that match a given naming pattern | This functionality should be added as a command line option to the main app in the spring-xd-ec2 project | 4 |
| 1138 | XD-1238 | Add to Acceptance Test EC2 CI build plan a stage that stops any existing CI EC2 deployments | Add a stage to the plan that will stop any instances that the CI process may have started before and relaunch a 2 node install based on rabbit. https://build.springsource.org/browse/XD-ATEC2 was created as an empty shell. The running of across different transports will be handled in a separate story along with adding a stage to run a 'hello world' acceptance test. | 8 |
| 1139 | XD-1239 | Add stage to Acceptance Test EC2 build plan that runs a basic acceptance test application against the single-node deployment | Run test application developed in XD-1245 | 2 |
| 1140 | XD-1240 | Add to Acceptance Test EC2 CI build plan a stage that uses XD distributed mode with rabbit | See https://quickstart.atlassian.com/download/bamboo/get-started/bamboo-elements "Stages are comprised of one or more Jobs, which run in parallel" we would like the tests across the rabbit and redis transport to occur in parallel. | 8 |
| 1141 | XD-1241 | Add to Acceptance Test EC2 job a stage that uses XD distributed mode with redis | See https://quickstart.atlassian.com/download/bamboo/get-started/bamboo-elements "Stages are comprised of one or more Jobs, which run in parallel" we would like the tests across the rabbit and redis transport to occur in parallel. | 8 |
| 1142 | XD-1242 | Optimize deployment of xd-admin and multiple nodes in spring-xd-ec2 to occur in parallel. | The deployment of nodes is sequential, we can reduce the time to deploy a cluster greatly by having these tasks execute in parallel. | 6 |
| 1143 | XD-1243 | Create App for CI to Shutdown XD EC2 Cluster | Application will shutdown all servers with a specific name. Application will take a --cluster-name parameter. Generate artifact to state success or failure | 5 |
| 1144 | XD-1244 | Create CI Plan for XD EC2 deployment | * Use the cleanup app from XD-1243 to stop previous CI runs of XD on EC2. * Build XD-EC2 deployer from github. * Use XD-EC2 Deployer to deploy the CI XD-Instance * Should produce artifact that contains the URL * admin server of the XD cluster. * container servers of the XD cluster | 8 |
| 1145 | XD-1245 | Develop basic acceptance test application to exercise based XD-EC2 deployment from CI | Create a first pass at an acceptance test app for a stream definition of http | log. This will involve creating two new projects in xd 1. spring-xd-integration-test 2. spring-xd-acceptance-tests #1 will contain generally useful utility methods for acceptance test, such as sending data over http, obtaining and asserting JMX values of specific modules. #2 will contain tests that use #1 to test the various out of the box modules provides in XD. | 5 |
| 1146 | XD-1246 | To be able to run the tests without conflicting with an existing XD admin server/launcher | In line with what we address at https://jira.springsource.org/browse/XD-1223, there are cases where tests fail when there is an existing instance of hsqldb running. Since hsqldb uses the same port and database, it causes issues. | 3 |
| 1147 | XD-1247 | Remove RedisStreamDeploymentIntegrationTests and RabbitStreamDeploymentIntegrationTests | These are duplicates of *SingleNodeDeploymentIntegrationTests | 1 |
| 1148 | XD-1248 | Merge AbstractStreamDeploymentIntegrationTests and AbstractSingleNodeStreamDeploymentIntegrationTests | We do not need two base classes for this | 2 |
| 1149 | XD-1249 | AbstractShellIntegrationTests should start and stop server once. | e.g., make application static and check for initialization. Need to ensure each test restores the initial state of the server | 5 |
| 1150 | XD-1250 | Investigate RabbitSingleNodeStreamDeploymentIntegrationTests performance | This test is very slow (2x the Redis version). Lots of stacktraces when running. Could be related. | 5 |
| 1151 | XD-1251 | Access-Control-Allow-Origin header should not be hard-coded | See also XD-451 as reference. | 4 |
| 1152 | XD-1252 | Allow processor script variables to be passed as module parameters | Currently, if we want to bind values to script variables we need to put them in a properties file like so: xd:> stream create --name groovyprocessortest --definition "http --port=9006 | script --location=custom-processor.groovy --properties-location=custom-processor.properties | log Ideally it should be: xd:> stream create --name groovyprocessortest --definition "http --port=9006 | script --location=custom-processor.groovy --foo=bar --baz=boo | log | 5 |
| 1153 | XD-1253 | RabbitMessageBus queue name prefix | I see the topics/exchanges created for Redis/Rabbit message buses have the prefix "topic." Is there any reason why we didn't have the prefix "queue." for the name of the queue created in Rabbit message bus in compared with the queue created in Redis message bus which has prefix "queue." | 1 |
| 1154 | XD-1254 | Optimize AbstractSingleNodeStreamDeploymentIntegrationTests | XD singlenode currenly initialized in @Before, should be @BeforeClass. In this case must be re-initialized for each transport, but not for each @Test. | 2 |
| 1155 | XD-1255 | Create assertion to get count of messages processed by a specific module in a stream | The modules are exposed via JMX and in turn exposed over http via jolokia. See https://jira.springsource.org/browse/XD-343. This issue is to develop a helper method that given a stream id and/or module name, assert that the number of messages processed after sending stimulus messages is as expected. e.g. int originalCount = getCount("testStream", "file"); //do stuff that generates 100 messages assertCount("testStream", "file", 100, originalCount) For now we can assume we know the location of where the modules are located by assuming we have only one container deployed. | 5 |
| 1156 | XD-1256 | Running XD as service | It is useful to configure operating system so that it will start Spring XD automatically on boot. For example, in Linux it would be great if Spring XD distro contains init.d script to run it as service. A typical init.d script gets executed with arguments such as "start", "stop", "restart", "pause", etc. In order for an init.d script to be started or stopped by init during startup and shutdown, the script needs to handle at least "start" and "stop" arguments. | 2 |
| 1157 | XD-1261 | Support default values for options derived out of ${} placeholders | The logic can be found in DefaultModuleOptionsMetadataCollector but caused problems in the initial PR. Revisit if needed | 4 |
| 1158 | XD-1262 | Provide a clean way to get a reference to the MessageBus running in SingleNodeApplication | Currently the message bus is only obtained via Module.getComponent(MessageBus.class). Stream testing scenarios that depend on sending and receiving payloads via named channels do not require a deployed module instance per se, but any stream flow control uses the MessageBus directly. Getting a deployed module instance in general is expensive, e.g., you have to wait for the module to deploy asynchronously, whereas the MessageBus implementation could be known a priori when the application starts. An improvement would be to ask the container for its MessageBus. | 3 |
| 1159 | XD-1263 | Copy latest build to S3 after a XD Build | NULL | 5 |
| 1160 | XD-1264 | Update SI to latest 4.0 M3 and Spring AMQP to 1.3.0.M2 | The Rabbit endpoint suffers from a problem similar to XD-1067. Seems like spring-[rabbit/amqp] needs to be bumped to 1.3.0.M1 to fix it. Sadly, we get this error: noformat java.lang.NoSuchMethodError: org.springframework.amqp.core.MessageProperties.getContentLength()J at org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper.extractStandardHeaders(DefaultAmqpHeaderMapper.java:102) at org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper.extractStandardHeaders(DefaultAmqpHeaderMapper.java:53) at org.springframework.integration.mapping.AbstractHeaderMapper.toHeaders(AbstractHeaderMapper.java:205) at org.springframework.integration.mapping.AbstractHeaderMapper.toHeadersFromRequest(AbstractHeaderMapper.java:148) at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$1.onMessage(AmqpInboundChannelAdapter.java:75) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:584) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:482) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:69) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:144) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:920) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:454) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:728) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:712) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$400(SimpleMessageListenerContainer.java:69) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:812) at java.lang.Thread.run(Thread.java:724) noformat Updating to latest SI snapshot does not help (as of Jan 23rd) | 4 |
| 1161 | XD-1265 | Surface better exception information to client | In trying to upgrade to latest SI, I encountered a failing test because it expects an error message to contain something, but SI changes make it disappear (the problem is an SI exception now has an explicit message and so does not expose its cause message anymore) This, however is the manifest of a deeper "problem". We currently expose the getMessage() of any generic Exception caught, in the VndError REST construct. But this is not enough. Things that we can consider are: 1) adding the whole stacktrace of the caught exception, as a String. This is not very good at it leaks java specific details 2) unwrap the caught exception to get to the deepest cause. This may not be what we want everytime 3) construct a VndErrors (note the 's') made of each layered exception 4) similar to 1), but not using the stacktrace, only the messages of each cause etc | 7 |
| 1162 | XD-1266 | Twitterstream is broken | NULL | 3 |
| 1163 | XD-1268 | Remove unused code related to 'accepted media type' in MessageBus | NULL | 4 |
| 1164 | XD-1269 | Investigate and remove explicit thread sleep in AbstractSingleNodeStreamDeploymentIntegrationTests | There are explicit Thread.sleep() calls after deploy() in some of the test methods in AbstractSingleNodeStreamDeploymentIntegrationTests. Also, the test method deployAndUndeploy() doesn't have explicit Thread.sleep() and fails inconsistently with this error: java.lang.AssertionError: stream test0 not undeployed org.junit.Assert.fail(Assert.java:88) org.junit.Assert.assertTrue(Assert.java:41) org.springframework.xd.dirt.stream.AbstractSingleNodeStreamDeploymentIntegrationTests.waitForUndeploy(AbstractSingleNodeStreamDeploymentIntegrationTests.java:332) org.springframework.xd.dirt.stream.AbstractSingleNodeStreamDeploymentIntegrationTests.deployAndUndeploy(AbstractSingleNodeStreamDeploymentIntegrationTests.java:221) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:601) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.springframework.xd.test.AbstractExternalResourceTestSupport$1.evaluate(AbstractExternalResourceTestSupport.java:71) org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) org.junit.rules.RunRules.evaluate(RunRules.java:20) org.junit.runners.ParentRunner.run(ParentRunner.java:309) org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:601) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) com.sun.proxy.$Proxy2.processTestClass(Unknown Source) org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:103) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:601) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355) org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:722) | 3 |
| 1165 | XD-1270 | Add states to the deployment of stream | Improve how the state of the stream is managed. A deploy command moves the stream from the undeployed state to the deploying state. If all modules in the stream are successfully deployed, the stream state is ‘deployed’ If one or more module deployments failed, the stream state is failed. Any modules that were successfully deployed, are still running. Sending an undeploy command will stop all modules of the stream and return the stream to the undeployed state. For the individual modules that failed, we will be able to find out which ones failed. Not yet sure if we can try to redeploy just those parts of the stream that failed. See the [design doc|https://docs.google.com/a/gopivotal.com/document/d/1kWtoH_xEF1wMklzQ8AZaiuhBZWIlpCDi8G9_hAP8Fgc/edit#heading=h.2rk74f16ow4i] for more details. Story points for this issue are the total of all the story points for the subtasks. | 20 |
| 1166 | XD-1271 | Investigate missing boot's actuator endpoints in XD | Currently, few of the boot's actuator endpoints go missing in the EndpointHandler mapping. They are: BeansEndpoint, dumpEndpoint, traceEndpoint, healthEndpoint, infoEndpoint. Also, the EndpointHandler mapping doesn't even happen in case of LauncherApplication. I think this is because the LauncherApplication context starts with port '0' and the TomcatEmbeddedServletContainer sets the local port for it later. With port '0', the Endpointhandler mapping is disabled during the EndpointHandler mapping bean creation. | 3 |
| 1167 | XD-1272 | Remove job options that are handled at module level from shell | Since makeUnique, dateFormat and numberFormat now have their own module options now, they are nicely advertised as options to a job (and will benefit from code completion soon), so they can be removed from the shell (where they currently allow for a misconfiguration if set at both the job and shell level) | 4 |
| 1168 | XD-1273 | The use of labelled modules and taps needs more explanation | https://github.com/spring-projects/spring-xd/wiki/Taps mentions this but the explanation needs more elaboration and example, e.g. mystream -> "http | flibble: transform --expression=payload.toUpperCase() | file" "tap:stream:mystream.flibble > transform --expression=payload.replaceAll('A','.') | log"); | 1 |
| 1169 | XD-1274 | spring-integration-hadoop.xsd (and reactor) Imports the SI 3.0 Instead of SI 4.0 Schema | NULL | 1 |
| 1170 | XD-1276 | Out of the box batch jobs should add xdJobExecutionListener and xdStepExecutionListener | To show best practice, our batch jobs should include these listeners so that notifications can be sent. In particular it is desirable that jobs with file item readers can send the files that were processed so another stream can be sent the file name when the job completes to move/delete the file | 8 |
| 1171 | XD-1277 | Default option values broken for composed modules | May want to fix "properly" when tackling the module options for composed modules, but this is currently broken (and wasn't at some point): module compose upperHttp --definition "http | transform --expression=payload.toUpperCase()" stream create foo --definition "upperHttp | log" This will fail saying that ${port} can't be resolved This will work though: module compose upperHttp --definition "http --port=9000 | transform --expression=payload.toUpperCase()" stream create foo --definition "upperHttp | log" Note that stream create foo --definition "upperHttp --port=xxx | log" should work too wut won't, but that's another bug (will create after this one) | 4 |
| 1172 | XD-1278 | Rename avro sink to hdfs-dataset and add support for parquet format | The kite SDK is used to write Avro records in a Kite specific format and also it support the parquet format for which we will eventually support. | 2 |
| 1173 | XD-1279 | The HDFS Sink should roll over based on the number of events. | NULL | 4 |
| 1174 | XD-1280 | JMX endpoints not functioning | I cannot access the JMX/Jolokia endpoints using the Spring Boot RC1 and Snapshot. Works with XD-M5 and Boot-M7 | 3 |
| 1175 | XD-1281 | Create Better UI instead of Boot's default "Whitelabel Error Page" | Create a better UI instead of Boot's default "Whitelabel Error Page" | 6 |
| 1176 | XD-1282 | Add caching to ModuleOptionsMetadataResolver | Will likely involve having the module identity (type+name) be part of the OptionsMetadata identity/cache key | 5 |
| 1177 | XD-1283 | Allow for late-binding of module options defaults | This is about computing the value to support expressions such as ${xd.stream.name} as a default. Initial discussion suggested to leverage the work done in XD-1175 by having a custom @LateValue (or @DeployTimeValue, etc) be resolved at deployment time | 4 |
| 1178 | XD-1284 | Support shell completions for closed set of values in stream definitions | When doing dsl completion in a stream definition and a module option accepts a value that has a closed set of possible values (eg booleans, enums), we can provide completions for those. | 8 |
| 1179 | XD-1285 | Support shell completions for module names | see CompletionProviderTests#testUnfinishedModuleNameShouldReturnCompletions() Ideally, would require a change in the parser so that it knows which kind of module was expected when it failed. | 4 |
| 1180 | XD-1286 | Use dedicated modules for code completion fingerprinting | see discussion at https://github.com/spring-projects/spring-xd/pull/495#discussion-diff-9291037 | 6 |
| 1181 | XD-1287 | Tap definitions should verify stream name | xd:>stream create --name simple --definition "http | transform | filter | transform | file" Created new stream 'simple' xd:>stream create --name tapSimple --definition "tap:stream:mystream.transform > file" Created new stream 'tapSimple' There isn't a stream named "mystream"... I don't remember if we want to allow for this (set up taps before there is a stream) or if it should be an error. Otherwise, works as expected xd:>stream create --name tapSimple2 --definition "tap:stream:simple.transform > file" Command failed org.springframework.xd.rest.client.impl.SpringXDException: XD144E:(pos 11): Reference to 'transform' is not unique in the target stream 'http | transform | filter | transform | file', please label the relevant module and use the label, or use a suffix index to indicate which occurrence of the module, e.g. 'transform.0' tap:simple.transform | 4 |
| 1182 | XD-1288 | Remove references to XD-1050 in documentation | NULL | 2 |
| 1183 | XD-1289 | Use descriptive texts for some module options defaults | Need a way to tell the user that this option will be determined at runtime,late bindings. In the module info command, references to ${xd.stream.name} could read "<use stream name>" for example) | 4 |
| 1184 | XD-1290 | Module context PropertyPlaceholderAutoConfiguration should have allowNulls = true | The current configuration prevents modules to have default values that evaluate to null. The workaround is to either: - have the module have its own PPC (which allows nulls) - rid the placeholders with ${foo:} | 4 |
| 1185 | XD-1291 | Handle container shutdown gracefully | Currently, ModuleDeployer is a disposable bean. When the container context is closed, the ModuleDeployer bean is destroyed along with its associated common context and deployed modules. Issue arises, if the connectionfactory bean associated with the deployed modules' message bus bindings is destroyed before the ModuleDeployer bean, there is exception stacktrace (at least in case of Redis MessageBus) saying "Connection closed". Adding SmartLifecycle support to ModuleDeployer will make sure all the beans are destroyed during Lifecycle processor's stop() method before any of the singletonbeans are destroyed. The stacktrace (when using Redis MessageBus is): org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is com.lambdaworks.redis.RedisException: Connection closed at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:45) at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:35) at org.springframework.data.redis.connection.lettuce.LettuceConverters.toDataAccessException(LettuceConverters.java:158) at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:237) at org.springframework.data.redis.connection.lettuce.LettuceConnection.bRPop(LettuceConnection.java:1449) at org.springframework.data.redis.core.DefaultListOperations$12.inRedis(DefaultListOperations.java:154) at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:50) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:181) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:149) at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84) at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:151) at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92) at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:178) at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:51) at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:291) at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) at java.lang.Thread.run(Thread.java:722) Caused by: com.lambdaworks.redis.RedisException: Connection closed at com.lambdaworks.redis.RedisAsyncConnection.await(RedisAsyncConnection.java:1079) at com.lambdaworks.redis.RedisConnection.await(RedisConnection.java:820) at com.lambdaworks.redis.RedisConnection.brpop(RedisConnection.java:101) at org.springframework.data.redis.connection.lettuce.LettuceConnection.bRPop(LettuceConnection.java:1447) ... 12 more 13:19:00,897 WARN Thread-5 support.DefaultLifecycleProcessor:257 - Failed to stop bean 'org.springframework.integration.redis.inbound.RedisInboundChannelAdapter#0' com.lambdaworks.redis.RedisException: Connection is closed at com.lambdaworks.redis.RedisAsyncConnection.dispatch(RedisAsyncConnection.java:1065) at com.lambdaworks.redis.pubsub.RedisPubSubConnection.unsubscribe(RedisPubSubConnection.java:82) at org.springframework.data.redis.connection.lettuce.LettuceSubscription.doUnsubscribe(LettuceSubscription.java:68) at org.springframework.data.redis.connection.util.AbstractSubscription.unsubscribe(AbstractSubscription.java:186) at org.springframework.data.redis.connection.util.AbstractSubscription.unsubscribe(AbstractSubscription.java:146) at org.springframework.data.redis.listener.RedisMessageListenerContainer$SubscriptionTask.cancel(RedisMessageListenerContainer.java:836) at org.springframework.data.redis.listener.RedisMessageListenerContainer.stop(RedisMessageListenerContainer.java:210) at org.springframework.integration.redis.inbound.RedisInboundChannelAdapter.doStop(RedisInboundChannelAdapter.java:127) at org.springframework.integration.endpoint.AbstractEndpoint.stop(AbstractEndpoint.java:100) at org.springframework.integration.endpoint.AbstractEndpoint.stop(AbstractEndpoint.java:115) at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:229) at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:51) at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:363) at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:202) at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:118) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:888) at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.doClose(EmbeddedWebApplicationContext.java:157) at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:809) | 2 |
| 1186 | XD-1292 | module delete command should only provide completions with composed modules | There is no point in providing completion with something that will fail when the user tries it. The information about a module being a composed is now available at the REST layer, so should be used | 4 |
| 1187 | XD-1293 | Remove the constraint on job module batch job id to be "job" | Currently, the job module's batch job's bean id should be "job". This also causes the job name to be 'actual-job-name + ".job"' and the batch job controllers require to search for job with suffix ".job". Removing this constraint would help us avoiding these. | 3 |
| 1188 | XD-1294 | Update spring-xd-extension-reactor dependency | Currently, the reactorEnv bean is defined in module-common context and the spring-xd-dirt has the runtime dependency over spring-xd-extension-reactor project. This enables boot's ReactorAutoConfiguration to initialize the reactor environment, we have the reactor setup configured for both admin and container server applications. Since reactor environment is not being used by container and only used by the reactor-syslog module, we can move the reactorEnv bean definition in reactor-syslog module. There is one caveat in this approach as the reactor environment gets setup everytime a new reactor-syslog module is deployed. | 1 |
| 1189 | XD-1295 | Module message conversion fails to work if JMX is enabled | If JMX is enabled for modules (enabling the IntegrationMBeanExporter), the ModuleTypeConversionPlugin fails to get the reference to input/output channel (for adding the ContentTypeHeaderInterceptor) and results in exception. It seems like the IntegrationMBeanExporter (when JMX is enabled) creates JdkDynamicAopProxy for all the integration components and thereby the following check on ModuleTypeConversionPlugin to retrieve the AbstractMessageChannel fails. AbstractMessageChannel channel = null; if (isInput) { channel = module.getComponent("input", AbstractMessageChannel.class); } else { channel = module.getComponent("output", AbstractMessageChannel.class); } I see the reason why AbstractMessageChannel is used here (to use some of the methods in the implementing class that didn't exist in the interfaces) but IntegrationMBeanExporter creating JdkDynamicAopProxy for the channel fails to resolve as AbstractMessageChannel here. Following is the full stack trace: To replicate, stream create testing --definition ""http --outputType=text/plain | log" 3:45,238 ERROR redisInboundAdapter-redis:queue-inbound-channel-adapter1 handler.LoggingHandler:145 - org.springframework.messaging.MessageHandlingException: error occurred in message handler [moduleDeployer] org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:79) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:601) org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) org.springframework.integration.monitor.SimpleMessageHandlerMetrics.handleMessage(SimpleMessageHandlerMetrics.java:106) org.springframework.integration.monitor.SimpleMessageHandlerMetrics.invoke(SimpleMessageHandlerMetrics.java:86) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) com.sun.proxy.$Proxy62.handleMessage(Unknown Source) org.springframework.integration.config.ServiceActivatorFactoryBean$1.handleRequestMessage(ServiceActivatorFactoryBean.java:83) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:152) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:121) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:108) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:218) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:188) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:601) org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) org.springframework.integration.monitor.DirectChannelMetrics.monitorSend(DirectChannelMetrics.java:113) org.springframework.integration.monitor.DirectChannelMetrics.doInvoke(DirectChannelMetrics.java:97) org.springframework.integration.monitor.DirectChannelMetrics.invoke(DirectChannelMetrics.java:91) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) com.sun.proxy.$Proxy61.send(Unknown Source) org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:114) org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:44) org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:92) org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:96) org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:212) org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:51) org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:291) org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52) java.lang.Thread.run(Thread.java:722) Caused by: org.springframework.xd.dirt.plugins.ModuleConfigurationException: Bean named 'output' must be of type [org.springframework.integration.channel.AbstractMessageChannel], but was actually of type [com.sun.proxy.$Proxy70] org.springframework.xd.dirt.plugins.stream.ModuleTypeConversionPlugin.configureModuleConverters(ModuleTypeConversionPlugin.java:144) org.springframework.xd.dirt.plugins.stream.ModuleTypeConversionPlugin.postProcessModule(ModuleTypeConversionPlugin.java:70) org.springframework.xd.dirt.module.ModuleDeployer.postProcessModule(ModuleDeployer.java:378) org.springframework.xd.dirt.module.ModuleDeployer.deploy(ModuleDeployer.java:282) org.springframework.xd.dirt.module.ModuleDeployer.deployAndStore(ModuleDeployer.java:271) org.springframework.xd.dirt.module.ModuleDeployer.handleDeploy(ModuleDeployer.java:266) org.springframework.xd.dirt.module.ModuleDeployer.handleSingleModuleMessage(ModuleDeployer.java:244) org.springframework.xd.dirt.module.ModuleDeployer.handleMessageInternal(ModuleDeployer.java:171) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) ... 42 more Caused by: org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'output' must be of type [org.springframework.integration.channel.AbstractMessageChannel], but was actually of type [com.sun.proxy.$Proxy70] org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:376) org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:979) org.springframework.xd.module.core.SimpleModule.getComponent(SimpleModule.java:156) org.springframework.xd.dirt.plugins.stream.ModuleTypeConversionPlugin.configureModuleConverters(ModuleTypeConversionPlugin.java:100) ... 50 more | 3 |
| 1190 | XD-1296 | Few integration tests fail if JMX is enabled | If JMX is enabled, some of the integration tests fail. This is similar to what we see in XD-1295. One example of this case is, the test classes that extend StreamTestSupport. In StreamTestSupport, the @BeforeClass has this line: moduleDeployer = containerContext.getBean(ModuleDeployer.class); When JMX is enable, the IntegrationMBeanExporter creates JdkDynamicProxy for the ModuleDeployer (since it is of type MessageHandler) and thereby the above line to get bean by the implementing class type (ModuleDeployer) fails. There are few other places where we use to refer the implementing classes on getBean(). Looks like we need to fix those as well. | 2 |
| 1191 | XD-1297 | Fix module type guessing heuristics | Commit 96de9fcfaf32719413015a1a6bace1b30b6b9610 strengthened module type inference, but some corner cases remain (marked as TODOs and commented out assertions in tests). To be effective, we need to look at the whole deployed stream (or composed module). Modify ParsinContext accordingly. | 5 |
| 1192 | XD-1298 | Commands that prompt the user are now broken | Most likely due to https://github.com/spring-projects/spring-shell/commit/296e4d2ff0e6e91d91209ab8717335357c587de0 When the user submits with ENTER, the shell appears to hang | 5 |
| 1193 | XD-1299 | Move --deleteFiles out of ResourcesIntoJdbcJobModuleOptionsMetadata | Currently, the fileDeletion listeners are added to filepollhdfs and filejdbc OOTB job modules so that the files are deleted after successful completion of jobs that write the file into hdfs/jdbc. We have "--deleteFiles" option in ResourcesIntoJdbcJobModuleOptionsMetadata (from https://github.com/spring-projects/spring-xd/pull/562) which makes it available for hdfsjdbc job module as well. But it is not supported yet as it involves deletion of HDFS files. We need the file deletion listeners for the hdfsjdbc and hdfsmongodb job modules so that if opted to delete files, it can be supported. | 3 |
| 1194 | XD-1300 | Handling boolean type module option properties defaults in option metadata | There are few boolean type module option properties whose default values are specified in the module definitions than their corresponding ModuleOptionsMetaData. Also, when using boolean we need to have module option using primitive type boolean than Boolean type. Currently, these are some of the module options that require this change: "initializeDatabase" in modules filejdbc, hdfsjdbc job modules, aggregator processor module, jdbc sink module "restartable" in all the job modules "deleteFiles" in filejdbc, filepollhdfs job modules | 3 |
| 1195 | XD-1301 | MBeans are not destroyed if stream is created and destroyed with no delay | Problem: The container that the stream was deployed to, will not allow new streams to be deployed. Once the error occurs, the only solution is to terminate the XD Container and restart it. To reproduce create a stream foo and destroy the stream, then create the stream foo again. This best done programmatically, taking the same steps using the "shell" may not reproduce the problem. i.e. if you put a Sleep of 1-2 seconds between the destroy and the next create, it works fine | 5 |
| 1196 | XD-1302 | Add documentation for using FTP->HDFS partitioned jobs | NULL | 5 |
| 1197 | XD-1303 | Create REST API for getting information on a given job instance | NULL | 3 |
| 1198 | XD-1304 | Create shell command for getting information on a given job instance | NULL | 3 |
| 1199 | XD-1305 | File Sink should support Replace as an option | currently the file sink only supports append. User should support an overwrite feature. | 5 |
| 1200 | XD-1306 | Move ftp support from .x package to spring-xd-dirt batch package | Commit https://github.com/spring-projects/spring-xd/commit/761cd5e8250c055878caf3a789ab5b3254ba48e8 introduced support for FTP and added a bunch of .x classes. These should not belong to DIRT proper though, and should be added to an extension style project. The job(s) module would then depend on them | 1 |
| 1201 | XD-1307 | Use HATEOAS Link templates | HATEOAS 0.9 introduced some support for templated links. This should be leveraged to properly handle eg /streams/{id} instead of using string concatenation | 5 |
| 1202 | XD-1308 | Weird behavior of the transform module | When trying some of the examples of XD-159, came up with weird behavior of the transform module. This boils down to: {noformat} stream create foo --definition "http |transform --expression='new java.lang.Integer(payload)' | transform --expression=payload.getClass() | log" http post --data 42 ==> Integer (OK) http post --data WTH => WTH (!) {noformat} Seems that when the expression can not be evaluated, the incoming payload is transmitted as is | 4 |
| 1203 | XD-1309 | JSR303 validation of options interferes with dsl completion | When using a JSR303 annotated class for module options, the binding failures should be bypassed, as they interfere with completion proposals. | 5 |
| 1204 | XD-1310 | Misleading error message when trying to restart a job exec | Disregard the missing date that is caused by another problem. Here is the setup: {noformat} xd:>job execution list Id Job Name Start Time Step Execution Count Status -- -------- -------------------------------- -------------------- --------- 13 foo Europe/Paris 0 STARTING 12 foo 2014-02-12 15:39:46 Europe/Paris 1 FAILED 11 foo 2014-02-12 15:39:29 Europe/Paris 1 COMPLETED 10 foo 2014-02-12 15:38:36 Europe/Paris 1 COMPLETED 9 foo 2014-02-12 15:38:21 Europe/Paris 1 COMPLETED 8 foo Europe/Paris 0 STARTING 7 foo 2014-02-12 15:25:41 Europe/Paris 1 COMPLETED 6 foo 2014-02-12 15:25:04 Europe/Paris 1 FAILED 5 foo 2014-02-12 15:14:32 Europe/Paris 1 FAILED 4 foo 2014-02-12 15:14:13 Europe/Paris 1 FAILED 3 foo 2014-02-12 15:13:54 Europe/Paris 1 FAILED 2 foo 2014-02-12 15:13:18 Europe/Paris 1 FAILED 1 foo 2014-02-12 15:12:58 Europe/Paris 1 FAILED 0 foo 2014-02-12 15:11:44 Europe/Paris 1 FAILED xd:>job execution restart --id 12 Command failed org.springframework.xd.rest.client.impl.SpringXDException: Job Execution 12 is already running. {noformat} while the server exception is a bit better: {noformat} Caused by: org.springframework.batch.core.repository.JobExecutionAlreadyRunningException: A job execution for this job is already running: JobInstance: id=11, version=0, Job=[foo] org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:120) {noformat} I'd argue we should not speak in terms of execution ids if possible, but rather in terms of job names | 1 |
| 1205 | XD-1311 | Job execution list should mention jobs that have been deleted | Create a job, execute it a couple of times, destroy it and then invoke job execution list. The job name column should mention that a job is defunct (even though a job with the same name could have been re-created in the interim) | 3 |
| 1206 | XD-1312 | Job execution restart fails with NPE | Create a job, launch it but make it fail (eg filejdbc with missing file) job execution list => it's there, as FAILED. Good job execution restart <theid> ==> Fails with NPE: {noformat} 16:59:42,160 ERROR http-nio-9393-exec-7 rest.RestControllerAdvice:191 - Caught exception while handling a request java.lang.NullPointerException org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:351) org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:117) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) com.sun.proxy.$Proxy39.run(Unknown Source) org.springframework.batch.admin.service.SimpleJobService.restart(SimpleJobService.java:179) org.springframework.xd.dirt.plugins.job.DistributedJobService.restart(DistributedJobService.java:77) org.springframework.xd.dirt.rest.BatchJobExecutionsController.restartJobExecution(BatchJobExecutionsController.java:146) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.springfram {noformat} | 5 |
| 1207 | XD-1314 | Create XD .zip distribution for YARN | Create XD .zip distribution for YARN that adds an additional sub-project to the spring-xd repo for building the xd-YARN.zip Link into main build file Produce a new artifact spring-xd-v-xyz-yarn.zip as part of the nightly CI process -- will now have 2 artifacts, main xd.zip distribution and xd-yarn.zip Does not include any Hadoop distribution libraries Does include spring-hadoop jars for Apache22 ‘unflavored’ | 3 |
| 1208 | XD-1315 | Create POJO based FileSink module metadata | This is so that we can have an ENUM that can show the possible values and autocomplete. Using just the XML the user has a greater chance to enter an invalid mode. | 5 |
| 1209 | XD-1316 | UI:Fix E2E test warning | When running E2E tests the following warning may be observed: {code} Running "karma:e2e" (karma) task INFO [karma]: Karma v0.10.9 server started at http://localhost:7070/_karma_/ INFO [launcher]: Starting browser PhantomJS TypeError: Cannot read property 'verbose' of undefined at enableWebsocket (/Users/hillert/dev/git/spring-xd/spring-xd-ui/node_modules/grunt-connect-proxy/lib/utils.js:101:18) at Object.utils.proxyRequest [as handle] (/Users/hillert/dev/git/spring-xd/spring-xd-ui/node_modules/grunt-connect-proxy/lib/utils.js:109:5) at next (/Users/hillert/dev/git/spring-xd/spring-xd-ui/node_modules/grunt-contrib-connect/node_modules/connect/lib/proto.js:193:15) at Object.livereload [as handle] (/Users/hillert/dev/git/spring-xd/spring-xd-ui/node_modules/grunt-contrib-connect/node_modules/connect-livereload/index.js:147:5) at next (/Users/hillert/dev/git/spring-xd/spring-xd-ui/node_modules/grunt-contrib-connect/node_modules/connect/lib/proto.js:193:15) at Function.app.handle (/Users/hillert/dev/git/spring-xd/spring-xd-ui/node_modules/grunt-contrib-connect/node_modules/connect/lib/proto.js:201:3) at Server.app (/Users/hillert/dev/git/spring-xd/spring-xd-ui/node_modules/grunt-contrib-connect/node_modules/connect/lib/connect.js:65:37) at Server.EventEmitter.emit (events.js:98:17) at HTTPParser.parser.onIncoming (http.js:2108:12) at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23) at Socket.socket.ondata (http.js:1966:22) at TCP.onread (net.js:525:27) {code} | 2 |
